[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:09 dist-test-slave-5l3k krb5kdc[2950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:09 dist-test-slave-5l3k krb5kdc[2950](info): set up 2 sockets
Aug 07 11:16:09 dist-test-slave-5l3k krb5kdc[2950](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 11:16:10.847342 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.009s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:10 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:11.040980 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36437
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:11.399042 2964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:11.399547 2964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:11.399752 2964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:11.408293 2964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:11.408484 2964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:11.408613 2964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:11.408738 2964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:11.408834 2964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:11.420181 2964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:11.422015 2964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:11.423838 2964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:11.445116 2971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:11.445555 2970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:11.445922 2973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:11.449618 2964 server_base.cc:1034] running on GCE node
I20260807 11:16:11.450841 2964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:11.452934 2964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:11.454144 2964 hybrid_clock.cc:648] HybridClock initialized: now 1786101371454050 us; error 116 us; skew 500 ppm
I20260807 11:16:11.454697 2964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:11 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101371, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:11.458391 2964 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:11.460814 2964 webserver.cc:460] Webserver started at http://127.2.223.190:44267/ using document root <none> and password file <none>
I20260807 11:16:11.461833 2964 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:11.461966 2964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:11.462374 2964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:11.467360 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "a2320686438d4219b937777f6f757d11"
format_stamp: "Formatted at 2026-08-07 11:16:11 on dist-test-slave-5l3k"
server_key: "750ea8e08396f9e670fc6ca107a4c11d"
server_key_iv: "cc08ceed7d4229dc3132114856b7f9fd"
server_key_version: "encryptionkey@0"
I20260807 11:16:11.468350 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "a2320686438d4219b937777f6f757d11"
format_stamp: "Formatted at 2026-08-07 11:16:11 on dist-test-slave-5l3k"
server_key: "750ea8e08396f9e670fc6ca107a4c11d"
server_key_iv: "cc08ceed7d4229dc3132114856b7f9fd"
server_key_version: "encryptionkey@0"
I20260807 11:16:11.475899 2964 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260807 11:16:11.481212 2980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:11.483270 2964 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260807 11:16:11.483516 2964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "a2320686438d4219b937777f6f757d11"
format_stamp: "Formatted at 2026-08-07 11:16:11 on dist-test-slave-5l3k"
server_key: "750ea8e08396f9e670fc6ca107a4c11d"
server_key_iv: "cc08ceed7d4229dc3132114856b7f9fd"
server_key_version: "encryptionkey@0"
I20260807 11:16:11.483716 2964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:11.504350 2964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:11.509358 2964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:11.509728 2964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:11.532352 2964 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:36437
I20260807 11:16:11.532377 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:36437 every 8 connection(s)
I20260807 11:16:11.534420 2964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:11.541210 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 2964
I20260807 11:16:11.541455 3037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:11.541632 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:11.545032 2942 external_mini_cluster.cc:1428] Setting key 5f2482caa9bcd3cc5ad6468b2d8eeb37
I20260807 11:16:11.561038 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11: Bootstrap starting.
I20260807 11:16:11.567109 3037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:11.569075 3037 log.cc:824] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:11.573804 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11: No bootstrap required, opened a new log
Aug 07 11:16:11 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:11.586019 3037 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } }
I20260807 11:16:11.586732 3037 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:11.586933 3037 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2320686438d4219b937777f6f757d11, State: Initialized, Role: FOLLOWER
I20260807 11:16:11.588111 3037 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } }
I20260807 11:16:11.588459 3037 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:11.588759 3037 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:11.589022 3037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:11.592410 3037 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } }
I20260807 11:16:11.593631 3037 leader_election.cc:302] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2320686438d4219b937777f6f757d11; no voters:
I20260807 11:16:11.594647 3037 leader_election.cc:288] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:11.594944 3042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:11.598439 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:11.552637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.2.223.190:36437)
0807 11:16:11.554628 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:11.554639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:11.554680 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:16:11.567441 (+ 12761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:11.567497 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:11.568051 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:11.568391 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:11.573209 (+ 4818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:11.574923 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:11.577761 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:11.578563 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:11.578790 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:11.585947 (+ 7157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:11.586026 (+ 79us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:11.586048 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:11.586121 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:11.590172 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:11.591431 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:11.591437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:11.591443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:11.591624 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:11.592193 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:11.592196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:11.592198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:11.592901 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:11.593222 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:11.595018 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0807 11:16:11.595601 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1695,"thread_start_us":1402,"threads_started":1}
I20260807 11:16:11.599390 3042 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [term 1 LEADER]: Becoming Leader. State: Replica: a2320686438d4219b937777f6f757d11, State: Running, Role: LEADER
I20260807 11:16:11.600456 3042 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } }
I20260807 11:16:11.601744 3037 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:11.606228 3044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2320686438d4219b937777f6f757d11. Latest consensus state: current_term: 1 leader_uuid: "a2320686438d4219b937777f6f757d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } } }
I20260807 11:16:11.606675 3044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:11.606338 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2320686438d4219b937777f6f757d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2320686438d4219b937777f6f757d11" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36437 } } }
I20260807 11:16:11.606892 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:11.607964 3050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:11.618115 3050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:11.633780 3050 catalog_manager.cc:1347] Generated new cluster ID: 44bcf7ddef704b6ab808fa0f93aa246c
I20260807 11:16:11.633898 3050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:11.646440 3050 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:11.648425 3050 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:11.663515 3050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2320686438d4219b937777f6f757d11: Generated new TSK 0
I20260807 11:16:11.664630 3050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:11.725803 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:12.319259 3066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:12.319617 3066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:12.319846 3066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:12.330302 3066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:12.330525 3066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:12.330749 3066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:12.344229 3066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:12.345778 3066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:12.347926 3066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:12.364537 3071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:12.365604 3072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:12.366309 3074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:12.366848 3066 server_base.cc:1034] running on GCE node
I20260807 11:16:12.367552 3066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:12.368566 3066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:12.369802 3066 hybrid_clock.cc:648] HybridClock initialized: now 1786101372369740 us; error 52 us; skew 500 ppm
I20260807 11:16:12.370215 3066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:12 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101372, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:12.374068 3066 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:12.376389 3066 webserver.cc:460] Webserver started at http://127.2.223.129:40695/ using document root <none> and password file <none>
I20260807 11:16:12.377347 3066 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:12.377597 3066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:12.378090 3066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:12.381722 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "d367d6d9a7c34890a33e7789de7d30c4"
format_stamp: "Formatted at 2026-08-07 11:16:12 on dist-test-slave-5l3k"
server_key: "3552446ba4c0791ccd80f67c1bb9f922"
server_key_iv: "9b0c67db146298ffb5c5c51c281a9192"
server_key_version: "encryptionkey@0"
I20260807 11:16:12.382745 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "d367d6d9a7c34890a33e7789de7d30c4"
format_stamp: "Formatted at 2026-08-07 11:16:12 on dist-test-slave-5l3k"
server_key: "3552446ba4c0791ccd80f67c1bb9f922"
server_key_iv: "9b0c67db146298ffb5c5c51c281a9192"
server_key_version: "encryptionkey@0"
I20260807 11:16:12.389982 3066 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260807 11:16:12.395218 3081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:12.396689 3066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:16:12.396921 3066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "d367d6d9a7c34890a33e7789de7d30c4"
format_stamp: "Formatted at 2026-08-07 11:16:12 on dist-test-slave-5l3k"
server_key: "3552446ba4c0791ccd80f67c1bb9f922"
server_key_iv: "9b0c67db146298ffb5c5c51c281a9192"
server_key_version: "encryptionkey@0"
I20260807 11:16:12.397210 3066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:12.413303 3066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:12.419394 3066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:12.419898 3066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:12.421655 3066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:12.423902 3066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:12.424083 3066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:12.424350 3066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:12.424453 3066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:12.525664 3066 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:41313
I20260807 11:16:12.525877 3201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:41313 every 8 connection(s)
I20260807 11:16:12.527576 3066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:12.530999 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3066
I20260807 11:16:12.531375 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:12.532730 2942 external_mini_cluster.cc:1428] Setting key 1f786e418eea5336e7aadc563193d308
Aug 07 11:16:12 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101372, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
I20260807 11:16:12.565393 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:12.533361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:49441 (local address 127.2.223.190:36437)
0807 11:16:12.543330 (+ 9969us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:12.543336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:12.543367 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:16:12.543441 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:12.543445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:12.543595 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:12.543731 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:12.546733 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:12.548024 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:12.550454 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:12.551195 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:12.551351 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:12.556562 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:12.556567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:12.556571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:12.556609 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:12.561174 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:12.562086 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:12.562090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:12.562092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:12.562227 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:12.562749 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:12.562761 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:12.562763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:12.563266 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:12.563599 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:12.564751 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0807 11:16:12.564994 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9752,"thread_start_us":9509,"threads_started":1}
I20260807 11:16:12.566805 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:12.532571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36437 (local address 127.2.223.129:49441)
0807 11:16:12.534198 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:12.534266 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:12.539057 (+ 4791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:12.544185 (+ 5128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:12.544200 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:12.545200 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:12.546468 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:12.546527 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:12.548246 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:12.548250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:12.549993 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:12.550005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:12.551602 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:12.551613 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:12.551949 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:12.552898 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:12.552921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:12.556270 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:12.561315 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:12.561324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:12.561340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:12.561943 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:12.562353 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:12.562364 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:12.562366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:12.562625 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:12.563570 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:12.563579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:12.563997 (+ 418us) client_negotiation.cc:769] Sending connection context
0807 11:16:12.564508 (+ 511us) client_negotiation.cc:240] Negotiation successful
0807 11:16:12.565137 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"thread_start_us":1041,"threads_started":1}
I20260807 11:16:12.568650 3202 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36437
I20260807 11:16:12.569139 3202 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:12.570320 3202 heartbeater.cc:511] Master 127.2.223.190:36437 requested a full tablet report, sending...
I20260807 11:16:12.574834 3000 ts_manager.cc:194] Registered new tserver with Master: d367d6d9a7c34890a33e7789de7d30c4 (127.2.223.129:41313)
I20260807 11:16:12.577504 3000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:49441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:12.593665 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:13.205768 3209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:13.206204 3209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:13.206458 3209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:13.215281 3209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:13.215463 3209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:13.215667 3209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:13.228411 3209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:13.230082 3209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:13.231631 3209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:13.252729 3214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:13.252966 3217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:13.253118 3215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:13.254956 3209 server_base.cc:1034] running on GCE node
I20260807 11:16:13.255656 3209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:13.256731 3209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:13.258036 3209 hybrid_clock.cc:648] HybridClock initialized: now 1786101373257946 us; error 32 us; skew 500 ppm
I20260807 11:16:13.258565 3209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:13 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101373, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:13.262796 3209 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:13.264784 3209 webserver.cc:460] Webserver started at http://127.2.223.130:38061/ using document root <none> and password file <none>
I20260807 11:16:13.265691 3209 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:13.265919 3209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:13.266297 3209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:13.269696 3209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "dcf9d13d3a514a7a8dc51a26a9d7761b"
format_stamp: "Formatted at 2026-08-07 11:16:13 on dist-test-slave-5l3k"
server_key: "be69928ccefd36d9b2d98ad91d75b1f0"
server_key_iv: "a098f4267bede95d837b82428369cae6"
server_key_version: "encryptionkey@0"
I20260807 11:16:13.270709 3209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "dcf9d13d3a514a7a8dc51a26a9d7761b"
format_stamp: "Formatted at 2026-08-07 11:16:13 on dist-test-slave-5l3k"
server_key: "be69928ccefd36d9b2d98ad91d75b1f0"
server_key_iv: "a098f4267bede95d837b82428369cae6"
server_key_version: "encryptionkey@0"
I20260807 11:16:13.277503 3209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260807 11:16:13.282219 3224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:13.283779 3209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:16:13.284016 3209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "dcf9d13d3a514a7a8dc51a26a9d7761b"
format_stamp: "Formatted at 2026-08-07 11:16:13 on dist-test-slave-5l3k"
server_key: "be69928ccefd36d9b2d98ad91d75b1f0"
server_key_iv: "a098f4267bede95d837b82428369cae6"
server_key_version: "encryptionkey@0"
I20260807 11:16:13.284308 3209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:13.314301 3209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:13.320406 3209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:13.320935 3209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:13.322333 3209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:13.324508 3209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:13.324694 3209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:13.324887 3209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:13.325003 3209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:13.409250 3209 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42511
I20260807 11:16:13.409369 3343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42511 every 8 connection(s)
I20260807 11:16:13.410737 3209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:13.417523 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3209
I20260807 11:16:13.417829 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:13.418510 2942 external_mini_cluster.cc:1428] Setting key 9443b8a6e4d71cf398f3a0f3375f9bda
Aug 07 11:16:13 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101373, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:13.437693 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:13.413790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:56279 (local address 127.2.223.190:36437)
0807 11:16:13.414601 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:13.414607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:13.415659 (+ 1052us) server_negotiation.cc:407] Connection header received
0807 11:16:13.418267 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:13.418273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:13.418360 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:13.418562 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:13.420621 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:13.421831 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:13.423984 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:13.425123 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:13.425267 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:13.430102 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:13.430108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:13.430111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:13.430144 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:13.433621 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:13.434729 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:13.434733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:13.434735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:13.434938 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:13.435649 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:13.435651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:13.435653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:13.435990 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:13.436129 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:13.437051 (+ 922us) server_negotiation.cc:299] Negotiation successful
0807 11:16:13.437227 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":542,"threads_started":1}
I20260807 11:16:13.439466 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:13.414233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36437 (local address 127.2.223.130:56279)
0807 11:16:13.415482 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:13.415544 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:13.417976 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:13.418796 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:13.418805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:13.419523 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:13.420402 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:13.420448 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:13.422063 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:13.422068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:13.423831 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:13.423844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:13.425394 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:13.425399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:13.425648 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:13.426395 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:13.426416 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:13.429844 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:13.433776 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:13.433787 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:13.433806 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:13.434612 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:13.435076 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:13.435084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:13.435089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:13.435497 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:13.436129 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:13.436140 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:13.436631 (+ 491us) client_negotiation.cc:769] Sending connection context
0807 11:16:13.436967 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 11:16:13.437593 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":683,"threads_started":1}
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:16:13.441159 3344 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36437
I20260807 11:16:13.441516 3344 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
I20260807 11:16:13.442756 3344 heartbeater.cc:511] Master 127.2.223.190:36437 requested a full tablet report, sending...
I20260807 11:16:13.444600 3000 ts_manager.cc:194] Registered new tserver with Master: dcf9d13d3a514a7a8dc51a26a9d7761b (127.2.223.130:42511)
I20260807 11:16:13.446239 3000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:56279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:13.471127 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:16:13.582635 3202 heartbeater.cc:503] Master 127.2.223.190:36437 was elected leader, sending a full tablet report...
W20260807 11:16:14.067101 3350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:14.067444 3350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:14.067684 3350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:14.077512 3350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:14.077682 3350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:14.077952 3350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:14.090168 3350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:14.091650 3350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:14.093346 3350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:14.110381 3355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:14.110536 3356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:14.111229 3358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:14.112017 3350 server_base.cc:1034] running on GCE node
I20260807 11:16:14.112656 3350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:14.113631 3350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:14.114957 3350 hybrid_clock.cc:648] HybridClock initialized: now 1786101374114833 us; error 50 us; skew 500 ppm
I20260807 11:16:14.115368 3350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:14.119596 3350 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:14.121665 3350 webserver.cc:460] Webserver started at http://127.2.223.131:36843/ using document root <none> and password file <none>
I20260807 11:16:14.122337 3350 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:14.122470 3350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:14.122879 3350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:14.125970 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "82cdee0626de49f3ad64dde2200c92a5"
format_stamp: "Formatted at 2026-08-07 11:16:14 on dist-test-slave-5l3k"
server_key: "fa3ee67732806472d226499d497b3e7b"
server_key_iv: "cfcd937f8b164266c966834cee00ce97"
server_key_version: "encryptionkey@0"
I20260807 11:16:14.126835 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "82cdee0626de49f3ad64dde2200c92a5"
format_stamp: "Formatted at 2026-08-07 11:16:14 on dist-test-slave-5l3k"
server_key: "fa3ee67732806472d226499d497b3e7b"
server_key_iv: "cfcd937f8b164266c966834cee00ce97"
server_key_version: "encryptionkey@0"
I20260807 11:16:14.133234 3350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 11:16:14.137658 3365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:14.139087 3350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:16:14.139233 3350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "82cdee0626de49f3ad64dde2200c92a5"
format_stamp: "Formatted at 2026-08-07 11:16:14 on dist-test-slave-5l3k"
server_key: "fa3ee67732806472d226499d497b3e7b"
server_key_iv: "cfcd937f8b164266c966834cee00ce97"
server_key_version: "encryptionkey@0"
I20260807 11:16:14.139395 3350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:14.167529 3350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:14.172349 3350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:14.172749 3350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:14.173995 3350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:14.175800 3350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:14.175912 3350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:14.176071 3350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:14.176205 3350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:14.253309 3350 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37463
I20260807 11:16:14.253459 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37463 every 8 connection(s)
I20260807 11:16:14.254944 3350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:14.255759 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3350
I20260807 11:16:14.256007 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:14.256773 2942 external_mini_cluster.cc:1428] Setting key d014cc5d18aa4e58f80c63b763511451
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:14.281106 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.258410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:54531 (local address 127.2.223.190:36437)
0807 11:16:14.259103 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.259110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.260629 (+ 1519us) server_negotiation.cc:407] Connection header received
0807 11:16:14.263753 (+ 3124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.263757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.263865 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.263975 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.265811 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.266886 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.268681 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.269282 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.269379 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.274011 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.274016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.274019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.274052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.278005 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.278861 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.278865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.278867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.279008 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.279474 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.279477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.279478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.279776 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.279912 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.280568 (+ 656us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.280739 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":445,"threads_started":1}
I20260807 11:16:14.282171 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.259200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36437 (local address 127.2.223.131:54531)
0807 11:16:14.260408 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:14.260465 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:14.263503 (+ 3038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:14.264173 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:14.264181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:14.264793 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:14.265659 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:14.265678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.267057 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.267061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:14.268560 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:14.268572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.269590 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.269596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:14.269846 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.270740 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:14.270762 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:14.273737 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:14.278168 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:14.278176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:14.278188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:14.278753 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:14.279123 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:14.279127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:14.279129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:14.279372 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:14.279892 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:14.279898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:14.280250 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 11:16:14.280479 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 11:16:14.280913 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":499,"threads_started":1}
I20260807 11:16:14.283617 3486 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36437
I20260807 11:16:14.283895 3486 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:14.284768 3486 heartbeater.cc:511] Master 127.2.223.190:36437 requested a full tablet report, sending...
I20260807 11:16:14.286383 3000 ts_manager.cc:194] Registered new tserver with Master: 82cdee0626de49f3ad64dde2200c92a5 (127.2.223.131:37463)
I20260807 11:16:14.287551 3000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:54531
I20260807 11:16:14.287809 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:14.342029 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.323690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.2.223.129:41313)
0807 11:16:14.324719 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.324727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.324753 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:14.324815 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.324822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.325026 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.325148 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.326827 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.327881 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.329371 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.329913 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.330008 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.334984 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.334996 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.335010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.335057 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.339066 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.339733 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.339740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.339742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.339942 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.340402 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.340406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.340408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.340795 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.341076 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.341380 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.341605 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":701,"threads_started":1}
W20260807 11:16:14.343685 3111 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45710
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:14.395525 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.380689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.2.223.129:41313)
0807 11:16:14.380964 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.380972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.381326 (+ 354us) server_negotiation.cc:407] Connection header received
0807 11:16:14.381540 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.381546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.381629 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.381739 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.383241 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.383969 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.385221 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.385675 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.385823 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.389974 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.389979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.389982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.390015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.392662 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.393301 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.393308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.393311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.393529 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.393899 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.393906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.393909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.394182 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.394531 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.394853 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.395128 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260807 11:16:14.397207 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3066
I20260807 11:16:14.409979 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.398291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.2.223.129:41313)
0807 11:16:14.398448 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.398452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.398883 (+ 431us) server_negotiation.cc:407] Connection header received
0807 11:16:14.399001 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.399004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.399065 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.399146 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.400233 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.400876 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.402346 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.402932 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.403023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.405016 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.405020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.405023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.405050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.407423 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.408127 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.408134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.408137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.408275 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.408646 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.408652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.408655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.408896 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.409160 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.409436 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.409642 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 11:16:14.410962 3197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:14.449800 3344 heartbeater.cc:503] Master 127.2.223.190:36437 was elected leader, sending a full tablet report...
I20260807 11:16:14.499502 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3066
I20260807 11:16:14.521298 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3209
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:14.541513 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.522772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.2.223.130:42511)
0807 11:16:14.523710 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.523718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.523744 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:14.523808 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.523815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.524144 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.524290 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.525447 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.527082 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.528333 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.529207 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.529328 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.534750 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.534763 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.534782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.534827 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.538579 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.539254 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.539263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.539267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.539418 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.539868 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.539873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.539877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.540296 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.540592 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.540861 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.541138 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":521,"threads_started":1}
I20260807 11:16:14.542958 3339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:14.626845 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3209
I20260807 11:16:14.647939 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3350
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[2950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:14.665349 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:14.649065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.2.223.131:37463)
0807 11:16:14.649658 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:14.649666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:14.649700 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:16:14.649775 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:14.649783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:14.650033 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:14.650251 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:14.651718 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.652783 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.654170 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:14.654895 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:14.655067 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:14.659640 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:14.659652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:14.659667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:14.659701 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:14.662424 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.663174 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.663182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.663184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.663302 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:14.663805 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:14.663809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:14.663811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:14.664190 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:14.664412 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:14.664740 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 11:16:14.664980 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":277,"threads_started":1}
I20260807 11:16:14.666702 3480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:14.749315 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3350
I20260807 11:16:14.770416 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 2964
I20260807 11:16:14.772064 3032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:14.879463 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 2964
2026-08-07T11:16:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5174 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[3509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[3509](info): set up 2 sockets
Aug 07 11:16:14 dist-test-slave-5l3k krb5kdc[3509](info): commencing operation
krb5kdc: starting...
W20260807 11:16:16.019594 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:16 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:16.218173 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40789
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:16.544689 3522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:16.544978 3522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:16.545161 3522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:16.552310 3522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:16.552412 3522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:16.552510 3522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:16.552613 3522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:16.552695 3522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:16.562310 3522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:16.563735 3522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:16.565045 3522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:16.576846 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:16.576874 3528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:16.576946 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:16.578039 3522 server_base.cc:1034] running on GCE node
I20260807 11:16:16.578879 3522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:16.580267 3522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:16.581478 3522 hybrid_clock.cc:648] HybridClock initialized: now 1786101376581430 us; error 35 us; skew 500 ppm
I20260807 11:16:16.581828 3522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:16 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:16.584978 3522 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:16.586469 3522 webserver.cc:460] Webserver started at http://127.2.223.190:38403/ using document root <none> and password file <none>
I20260807 11:16:16.587212 3522 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:16.587339 3522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:16.587667 3522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:16.590622 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "70e951942b164643899876f11d0ea916"
format_stamp: "Formatted at 2026-08-07 11:16:16 on dist-test-slave-5l3k"
server_key: "1e359096eb177812aa12b8605c17a756"
server_key_iv: "7a6224343a491495bb443435d0e2019e"
server_key_version: "encryptionkey@0"
I20260807 11:16:16.591380 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "70e951942b164643899876f11d0ea916"
format_stamp: "Formatted at 2026-08-07 11:16:16 on dist-test-slave-5l3k"
server_key: "1e359096eb177812aa12b8605c17a756"
server_key_iv: "7a6224343a491495bb443435d0e2019e"
server_key_version: "encryptionkey@0"
I20260807 11:16:16.596517 3522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260807 11:16:16.599946 3537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:16.601280 3522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:16:16.601462 3522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "70e951942b164643899876f11d0ea916"
format_stamp: "Formatted at 2026-08-07 11:16:16 on dist-test-slave-5l3k"
server_key: "1e359096eb177812aa12b8605c17a756"
server_key_iv: "7a6224343a491495bb443435d0e2019e"
server_key_version: "encryptionkey@0"
I20260807 11:16:16.601658 3522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:16.614773 3522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:16.618498 3522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:16.618798 3522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:16.636974 3522 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:40789
I20260807 11:16:16.637004 3590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:40789 every 8 connection(s)
I20260807 11:16:16.638315 3522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:16.641827 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:16.646397 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3522
I20260807 11:16:16.646664 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:16.647482 2942 external_mini_cluster.cc:1428] Setting key 341fbabcc13d52388038924a763d8d7c
I20260807 11:16:16.653200 3591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916: Bootstrap starting.
I20260807 11:16:16.656806 3591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:16.657996 3591 log.cc:824] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:16.661471 3591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916: No bootstrap required, opened a new log
Aug 07 11:16:16 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:16.672290 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:16.649473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.2.223.190:40789)
0807 11:16:16.650292 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:16.650304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:16.650343 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:16:16.652109 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:16.652130 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:16.652469 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:16.652804 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:16.654647 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:16.655931 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:16.657675 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:16.658391 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:16.658646 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:16.663309 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:16.663319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:16.663338 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:16.663381 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:16.667520 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:16.668350 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:16.668363 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:16.668372 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:16.668604 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:16.668942 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:16.668947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:16.668951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:16.669639 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:16.669952 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:16.670309 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 11:16:16.670829 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":384,"threads_started":1}
I20260807 11:16:16.672583 3591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } }
I20260807 11:16:16.673113 3591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:16.673240 3591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e951942b164643899876f11d0ea916, State: Initialized, Role: FOLLOWER
I20260807 11:16:16.673874 3591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } }
I20260807 11:16:16.674161 3591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:16.674299 3591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:16.674517 3591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:16.677057 3591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } }
I20260807 11:16:16.677537 3591 leader_election.cc:302] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e951942b164643899876f11d0ea916; no voters:
I20260807 11:16:16.678149 3591 leader_election.cc:288] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:16.678332 3596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:16.679410 3596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [term 1 LEADER]: Becoming Leader. State: Replica: 70e951942b164643899876f11d0ea916, State: Running, Role: LEADER
I20260807 11:16:16.679908 3596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } }
I20260807 11:16:16.680383 3591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:16.682713 3597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e951942b164643899876f11d0ea916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } } }
I20260807 11:16:16.683214 3597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:16.683362 3598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e951942b164643899876f11d0ea916. Latest consensus state: current_term: 1 leader_uuid: "70e951942b164643899876f11d0ea916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e951942b164643899876f11d0ea916" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40789 } } }
I20260807 11:16:16.684818 3598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:16.686996 3605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:16.692374 3605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:16.702191 3605 catalog_manager.cc:1347] Generated new cluster ID: 8d63df37cf144b7a96b5cc111b18fbec
I20260807 11:16:16.702337 3605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:16.716696 3605 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:16.718120 3605 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:16.724627 3605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e951942b164643899876f11d0ea916: Generated new TSK 0
I20260807 11:16:16.725605 3605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:16.779747 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:17.116482 3618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:17.116793 3618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:17.116986 3618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:17.124091 3618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:17.124254 3618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:17.124372 3618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:17.133770 3618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:17.134963 3618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:17.136234 3618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:17.149446 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:17.149540 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:17.149541 3624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:17.150385 3618 server_base.cc:1034] running on GCE node
I20260807 11:16:17.150982 3618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:17.151872 3618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:17.153075 3618 hybrid_clock.cc:648] HybridClock initialized: now 1786101377153039 us; error 28 us; skew 500 ppm
I20260807 11:16:17.153347 3618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:17 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101377, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:17.156471 3618 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:17.158041 3618 webserver.cc:460] Webserver started at http://127.2.223.129:45411/ using document root <none> and password file <none>
I20260807 11:16:17.158836 3618 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:17.158984 3618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:17.159302 3618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:17.162119 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "74fcf94d508f4971b950391f4e95b9e3"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "09e52fffc21ba09064744be23e9a32f8"
server_key_iv: "068354b711732cc2dce741af90d40d91"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.162920 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "74fcf94d508f4971b950391f4e95b9e3"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "09e52fffc21ba09064744be23e9a32f8"
server_key_iv: "068354b711732cc2dce741af90d40d91"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.167867 3618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 11:16:17.171394 3635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.172853 3618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:16:17.173022 3618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "74fcf94d508f4971b950391f4e95b9e3"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "09e52fffc21ba09064744be23e9a32f8"
server_key_iv: "068354b711732cc2dce741af90d40d91"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.173231 3618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:17.197237 3618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:17.201360 3618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:17.201674 3618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:17.202765 3618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:17.204121 3618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:17.204226 3618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.204397 3618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:17.204471 3618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.239629 3618 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:37417
I20260807 11:16:17.239718 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:37417 every 8 connection(s)
I20260807 11:16:17.240976 3618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:17.248696 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3618
I20260807 11:16:17.248936 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:17.249689 2942 external_mini_cluster.cc:1428] Setting key 23cf05d5e8318aba4e5e61c814b018d2
Aug 07 11:16:17 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101377, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:17.261898 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:17.243898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:34729 (local address 127.2.223.190:40789)
0807 11:16:17.244404 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:17.244412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:17.245362 (+ 950us) server_negotiation.cc:407] Connection header received
0807 11:16:17.247218 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:17.247223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:17.247289 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:17.247411 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:17.249357 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.250174 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.252106 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.252504 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.252652 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:17.256148 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:17.256153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:17.256157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:17.256188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:17.258906 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:17.259646 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:17.259657 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:17.259659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:17.259774 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:17.260210 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:17.260215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:17.260224 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:17.260468 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:17.260627 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:17.261335 (+ 708us) server_negotiation.cc:299] Negotiation successful
0807 11:16:17.261552 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260807 11:16:17.262918 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:17.244461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40789 (local address 127.2.223.129:34729)
0807 11:16:17.245201 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:17.245247 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:17.246968 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:17.247617 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:17.247624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:17.248295 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:17.249166 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:17.249195 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.250349 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.250353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:17.251956 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:17.251968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.252746 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.252754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:17.252984 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:17.253672 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:17.253693 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:17.255927 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:17.259051 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:17.259061 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:17.259077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:17.259506 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:17.259888 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:17.259893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:17.259897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:17.260086 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:17.260617 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:17.260624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:17.260941 (+ 317us) client_negotiation.cc:769] Sending connection context
0807 11:16:17.261200 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 11:16:17.261735 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":266,"threads_started":1}
I20260807 11:16:17.264645 3754 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40789
I20260807 11:16:17.264997 3754 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:17.265888 3754 heartbeater.cc:511] Master 127.2.223.190:40789 requested a full tablet report, sending...
I20260807 11:16:17.269026 3554 ts_manager.cc:194] Registered new tserver with Master: 74fcf94d508f4971b950391f4e95b9e3 (127.2.223.129:37417)
I20260807 11:16:17.271507 3554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:34729
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:17.303414 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:17.706938 3760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:17.707257 3760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:17.707492 3760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:17.716081 3760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:17.716219 3760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:17.716426 3760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:17.729125 3760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:17.730592 3760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:17.732187 3760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:17.750149 3765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:17.750509 3766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:17.751412 3768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:17.751781 3760 server_base.cc:1034] running on GCE node
I20260807 11:16:17.752609 3760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:17.753767 3760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:17.755071 3760 hybrid_clock.cc:648] HybridClock initialized: now 1786101377754971 us; error 44 us; skew 500 ppm
I20260807 11:16:17.755507 3760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:17 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101377, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:17.759496 3760 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:17.761126 3760 webserver.cc:460] Webserver started at http://127.2.223.130:45601/ using document root <none> and password file <none>
I20260807 11:16:17.762004 3760 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:17.762286 3760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:17.762796 3760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:17.766117 3760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "311e96806e4043d7a324a38e2fc7839f"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "f7b0950f0ca8d3fce8ce5b0639009a80"
server_key_iv: "5a1ce746b9635d846c086b7699e74c2c"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.767128 3760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "311e96806e4043d7a324a38e2fc7839f"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "f7b0950f0ca8d3fce8ce5b0639009a80"
server_key_iv: "5a1ce746b9635d846c086b7699e74c2c"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.774166 3760 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260807 11:16:17.779269 3777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.780809 3760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:17.781034 3760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "311e96806e4043d7a324a38e2fc7839f"
format_stamp: "Formatted at 2026-08-07 11:16:17 on dist-test-slave-5l3k"
server_key: "f7b0950f0ca8d3fce8ce5b0639009a80"
server_key_iv: "5a1ce746b9635d846c086b7699e74c2c"
server_key_version: "encryptionkey@0"
I20260807 11:16:17.781399 3760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:17.809698 3760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:17.815771 3760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:17.816191 3760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:17.817981 3760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:17.820286 3760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:17.820462 3760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.820729 3760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:17.820883 3760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:17.892851 3760 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:36713
I20260807 11:16:17.892895 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:36713 every 8 connection(s)
I20260807 11:16:17.894295 3760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:17.896591 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3760
I20260807 11:16:17.896871 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:17.897491 2942 external_mini_cluster.cc:1428] Setting key dd9abf252682f9d6c2e4712c132ab0aa
Aug 07 11:16:17 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101377, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:17.915036 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:17.898350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:34807 (local address 127.2.223.190:40789)
0807 11:16:17.898832 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:17.898839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:17.898954 (+ 115us) server_negotiation.cc:407] Connection header received
0807 11:16:17.900948 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:17.900954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:17.901053 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:17.901179 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:17.902691 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.903448 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.904973 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.905482 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.905619 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:17.909375 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:17.909380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:17.909383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:17.909413 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:17.912041 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:17.912778 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:17.912782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:17.912785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:17.912907 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:17.913349 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:17.913354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:17.913356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:17.913606 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:17.913764 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:17.914426 (+ 662us) server_negotiation.cc:299] Negotiation successful
0807 11:16:17.914667 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260807 11:16:17.915940 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:17.897757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40789 (local address 127.2.223.130:34807)
0807 11:16:17.898770 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:17.898832 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:17.900692 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:17.901405 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:17.901413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:17.901918 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:17.902466 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:17.902551 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.903602 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.903608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:17.904827 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:17.904839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:17.905714 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:17.905722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:17.905990 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:17.906791 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:17.906813 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:17.909152 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:17.912186 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:17.912194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:17.912210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:17.912643 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:17.913021 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:17.913029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:17.913032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:17.913220 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:17.913744 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:17.913752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:17.914058 (+ 306us) client_negotiation.cc:769] Sending connection context
0807 11:16:17.914328 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 11:16:17.914856 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":376,"threads_started":1}
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:16:17.917342 3897 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40789
I20260807 11:16:17.917591 3897 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:17.918330 3897 heartbeater.cc:511] Master 127.2.223.190:40789 requested a full tablet report, sending...
I20260807 11:16:17.919910 3554 ts_manager.cc:194] Registered new tserver with Master: 311e96806e4043d7a324a38e2fc7839f (127.2.223.130:36713)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
I20260807 11:16:17.920934 3554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:34807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:17.942363 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:18.267642 3903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:18.267899 3903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:18.267994 3903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:18.275179 3903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:18.275339 3903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260807 11:16:18.275350 3754 heartbeater.cc:503] Master 127.2.223.190:40789 was elected leader, sending a full tablet report...
W20260807 11:16:18.275563 3903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:18.285674 3903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:18.286996 3903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:18.288319 3903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:18.301527 3911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:18.301589 3909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:18.301612 3908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:18.302735 3903 server_base.cc:1034] running on GCE node
I20260807 11:16:18.303323 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:18.304234 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:18.305528 3903 hybrid_clock.cc:648] HybridClock initialized: now 1786101378305472 us; error 48 us; skew 500 ppm
I20260807 11:16:18.305859 3903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:18.309053 3903 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:18.310345 3903 webserver.cc:460] Webserver started at http://127.2.223.131:43599/ using document root <none> and password file <none>
I20260807 11:16:18.311031 3903 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:18.311131 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:18.311381 3903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:18.314172 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "2bee4e0b841a4abe98743cf59033573b"
format_stamp: "Formatted at 2026-08-07 11:16:18 on dist-test-slave-5l3k"
server_key: "6a6db1341c40666bd5d09034cd656fe3"
server_key_iv: "f68a90c6587d705acc992b149ed438bb"
server_key_version: "encryptionkey@0"
I20260807 11:16:18.314949 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "2bee4e0b841a4abe98743cf59033573b"
format_stamp: "Formatted at 2026-08-07 11:16:18 on dist-test-slave-5l3k"
server_key: "6a6db1341c40666bd5d09034cd656fe3"
server_key_iv: "f68a90c6587d705acc992b149ed438bb"
server_key_version: "encryptionkey@0"
I20260807 11:16:18.320418 3903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:18.323885 3919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:18.325259 3903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:18.325392 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "2bee4e0b841a4abe98743cf59033573b"
format_stamp: "Formatted at 2026-08-07 11:16:18 on dist-test-slave-5l3k"
server_key: "6a6db1341c40666bd5d09034cd656fe3"
server_key_iv: "f68a90c6587d705acc992b149ed438bb"
server_key_version: "encryptionkey@0"
I20260807 11:16:18.325568 3903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:18.352442 3903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:18.356900 3903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:18.357151 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:18.358103 3903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:18.359494 3903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:18.359577 3903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:18.359659 3903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:18.359704 3903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:18.393744 3903 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37239
I20260807 11:16:18.393783 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37239 every 8 connection(s)
I20260807 11:16:18.395064 3903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:18.399827 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 3903
I20260807 11:16:18.400030 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:18.400604 2942 external_mini_cluster.cc:1428] Setting key 40479b1e366a4c41fffaba1ee74f45c9
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:18.415694 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.397702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:47325 (local address 127.2.223.190:40789)
0807 11:16:18.397939 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.397943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.399237 (+ 1294us) server_negotiation.cc:407] Connection header received
0807 11:16:18.401737 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.401743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.401824 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.401939 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.403785 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.404669 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.406362 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.406797 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.406907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.410145 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.410150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.410153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.410186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.412661 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.413310 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.413314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.413316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.413426 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.413792 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.413795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.413796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.414038 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.414181 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.415064 (+ 883us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.415340 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:16:18.416707 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.398244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40789 (local address 127.2.223.131:47325)
0807 11:16:18.398987 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.399097 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.401416 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.402221 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.402233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.402996 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:18.403603 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.403631 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.404857 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.404861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.406220 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.406236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.407013 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.407018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.407211 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.407802 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:18.407824 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:18.409925 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:18.412784 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:18.412790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:18.412801 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:18.413210 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:18.413540 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:18.413543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:18.413545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:18.413705 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:18.414186 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:18.414194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:18.414651 (+ 457us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.414925 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.415379 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
I20260807 11:16:18.418015 4038 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40789
I20260807 11:16:18.418270 4038 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:18.418975 4038 heartbeater.cc:511] Master 127.2.223.190:40789 requested a full tablet report, sending...
I20260807 11:16:18.420255 3554 ts_manager.cc:194] Registered new tserver with Master: 2bee4e0b841a4abe98743cf59033573b (127.2.223.131:37239)
I20260807 11:16:18.420991 3554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:47325
I20260807 11:16:18.428340 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:16:18.447603 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.435156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.2.223.190:40789)
0807 11:16:18.435350 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.435355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.435842 (+ 487us) server_negotiation.cc:407] Connection header received
0807 11:16:18.436030 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.436034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.436107 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.436200 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.437671 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.438393 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.439772 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.440234 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.440338 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.442639 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.442645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.442648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.442679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.445162 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.445833 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.445837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.445839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.445949 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.446289 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.446292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.446294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.446668 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.446870 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.447222 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.447371 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:16:18.455819 3554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:18.463690 3554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:18.505982 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.479688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.2.223.129:37417)
0807 11:16:18.498424 (+ 18736us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.498432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.498460 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:16:18.498592 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.498600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.498817 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.499495 (+ 678us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:18.499658 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500617 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.503853 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.504804 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.504972 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.505229 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.505457 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.505618 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18554,"thread_start_us":19201,"threads_started":1}
I20260807 11:16:18.506182 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.479437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37417 (local address 127.0.0.1:49270)
0807 11:16:18.480294 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.480342 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.480656 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.499024 (+ 18368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.499028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.499052 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:18.499336 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.499345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500759 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.500767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.503711 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.503722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.505002 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.505008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.505196 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.505254 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.505414 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.505627 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":360,"thread_start_us":15256,"threads_started":1}
I20260807 11:16:18.507030 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.479440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:37239 (local address 127.0.0.1:45612)
0807 11:16:18.494992 (+ 15552us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.495014 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.495147 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.497701 (+ 2554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.497710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.497763 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:18.498765 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.498780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500814 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.500824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.503555 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.503571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.505931 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.505938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.506277 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.506314 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.506402 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.506554 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15410,"mutex_wait_us":26,"thread_start_us":904,"threads_started":1}
I20260807 11:16:18.507807 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.481678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.2.223.130:36713)
0807 11:16:18.497133 (+ 15455us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.497142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.497171 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:16:18.497259 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.497264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.497552 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.497787 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:18.498933 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500653 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.503510 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.505103 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.505263 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.507157 (+ 1894us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.507255 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.507438 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15268,"thread_start_us":15134,"threads_started":1}
I20260807 11:16:18.507973 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.480124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.2.223.131:37239)
0807 11:16:18.496837 (+ 16713us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.496850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.496896 (+ 46us) server_negotiation.cc:407] Connection header received
0807 11:16:18.497040 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.497052 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.497446 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.497676 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:18.498986 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500647 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.503733 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.505353 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.505516 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.505758 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.507390 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.507580 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16470,"thread_start_us":16260,"threads_started":1}
I20260807 11:16:18.507444 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.481365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:36713 (local address 127.0.0.1:50626)
0807 11:16:18.495903 (+ 14538us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.495925 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.496086 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.498086 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.498093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.498123 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:18.498598 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.498622 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.500843 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.500847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.503330 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.503346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.505629 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.505636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.506735 (+ 1099us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.506771 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.506849 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.507025 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14382,"mutex_wait_us":172,"thread_start_us":14208,"threads_started":1}
I20260807 11:16:18.511243 3684 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce5935820ff4aab98b1001d6c19e2c6 (DEFAULT_TABLE table=test-table [id=c704368ab9bc41489817b1a9b58514de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:18.511629 3828 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce5935820ff4aab98b1001d6c19e2c6 (DEFAULT_TABLE table=test-table [id=c704368ab9bc41489817b1a9b58514de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:18.511906 3969 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce5935820ff4aab98b1001d6c19e2c6 (DEFAULT_TABLE table=test-table [id=c704368ab9bc41489817b1a9b58514de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:18.513042 3828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce5935820ff4aab98b1001d6c19e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:18.513084 3684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce5935820ff4aab98b1001d6c19e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:18.513322 3969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce5935820ff4aab98b1001d6c19e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:18.528132 4057 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Bootstrap starting.
I20260807 11:16:18.532686 4057 tablet_bootstrap.cc:654] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:18.533380 4059 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Bootstrap starting.
I20260807 11:16:18.536487 4057 log.cc:824] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:18.540256 4059 tablet_bootstrap.cc:654] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:18.542301 4057 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: No bootstrap required, opened a new log
I20260807 11:16:18.542424 4059 log.cc:824] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:18.542979 4057 ts_tablet_manager.cc:1397] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260807 11:16:18.543022 4058 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Bootstrap starting.
I20260807 11:16:18.549063 4058 tablet_bootstrap.cc:654] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:18.549371 4059 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: No bootstrap required, opened a new log
I20260807 11:16:18.549710 4059 ts_tablet_manager.cc:1397] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260807 11:16:18.551065 4058 log.cc:824] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:18.554283 4058 tablet_bootstrap.cc:492] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: No bootstrap required, opened a new log
I20260807 11:16:18.554672 4058 ts_tablet_manager.cc:1397] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260807 11:16:18.558979 4057 raft_consensus.cc:348] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.559886 4057 raft_consensus.cc:374] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:18.560139 4057 raft_consensus.cc:729] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311e96806e4043d7a324a38e2fc7839f, State: Initialized, Role: FOLLOWER
I20260807 11:16:18.561087 4057 consensus_queue.cc:260] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.563004 3897 heartbeater.cc:503] Master 127.2.223.190:40789 was elected leader, sending a full tablet report...
I20260807 11:16:18.563807 4057 ts_tablet_manager.cc:1428] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
I20260807 11:16:18.564303 4059 raft_consensus.cc:348] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.564810 4059 raft_consensus.cc:374] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:18.564952 4059 raft_consensus.cc:729] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bee4e0b841a4abe98743cf59033573b, State: Initialized, Role: FOLLOWER
I20260807 11:16:18.565649 4059 consensus_queue.cc:260] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.567292 4038 heartbeater.cc:503] Master 127.2.223.190:40789 was elected leader, sending a full tablet report...
I20260807 11:16:18.567881 4059 ts_tablet_manager.cc:1428] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260807 11:16:18.569140 4058 raft_consensus.cc:348] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.569684 4058 raft_consensus.cc:374] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:18.569855 4058 raft_consensus.cc:729] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf94d508f4971b950391f4e95b9e3, State: Initialized, Role: FOLLOWER
I20260807 11:16:18.570544 4058 consensus_queue.cc:260] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.572990 4058 ts_tablet_manager.cc:1428] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
W20260807 11:16:18.646445 3898 tablet.cc:2369] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:16:18.646937 4039 tablet.cc:2369] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:18.657850 4063 raft_consensus.cc:482] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:18.658454 4063 raft_consensus.cc:504] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.661029 4063 leader_election.cc:288] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf94d508f4971b950391f4e95b9e3 (127.2.223.129:37417), 2bee4e0b841a4abe98743cf59033573b (127.2.223.131:37239)
I20260807 11:16:18.670426 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.661811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49501 (local address 127.2.223.131:37239)
0807 11:16:18.662139 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.662146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.663371 (+ 1225us) server_negotiation.cc:407] Connection header received
0807 11:16:18.663686 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.663690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.663750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.663855 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:18.664633 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.665559 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.668150 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.669216 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.669446 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.669553 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.669898 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.670025 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 11:16:18.670809 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.661429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:37239 (local address 127.2.223.130:49501)
0807 11:16:18.663217 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.663243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.663506 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.663877 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.663881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.663947 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:18.664477 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.664487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.665699 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.665705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.667981 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.667994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.669550 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.669558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.669720 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.669745 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.669858 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.669990 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"mutex_wait_us":1,"thread_start_us":948,"threads_started":1}
I20260807 11:16:18.671403 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.661787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:59151 (local address 127.2.223.129:37417)
0807 11:16:18.662171 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.662181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.663800 (+ 1619us) server_negotiation.cc:407] Connection header received
0807 11:16:18.664085 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.664089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.664157 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.664272 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:18.665241 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.666337 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.668789 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.669966 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.670209 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.670336 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.670921 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.671077 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260807 11:16:18.671536 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.661891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37417 (local address 127.2.223.130:59151)
0807 11:16:18.663673 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:18.663690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:18.663887 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:18.664291 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:18.664296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:18.664329 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:18.665047 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.665059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.666552 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.666557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.668598 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:18.668612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.670361 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.670371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:18.670715 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.670757 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 11:16:18.670891 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 11:16:18.671024 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"thread_start_us":1222,"threads_started":1}
I20260807 11:16:18.672351 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce5935820ff4aab98b1001d6c19e2c6" candidate_uuid: "311e96806e4043d7a324a38e2fc7839f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bee4e0b841a4abe98743cf59033573b" is_pre_election: true
I20260807 11:16:18.673168 3990 raft_consensus.cc:2454] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311e96806e4043d7a324a38e2fc7839f in term 0.
I20260807 11:16:18.673121 3705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce5935820ff4aab98b1001d6c19e2c6" candidate_uuid: "311e96806e4043d7a324a38e2fc7839f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf94d508f4971b950391f4e95b9e3" is_pre_election: true
I20260807 11:16:18.673802 3705 raft_consensus.cc:2454] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311e96806e4043d7a324a38e2fc7839f in term 0.
I20260807 11:16:18.674203 3781 leader_election.cc:302] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bee4e0b841a4abe98743cf59033573b, 311e96806e4043d7a324a38e2fc7839f; no voters:
I20260807 11:16:18.674803 4063 raft_consensus.cc:2781] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:18.675081 4063 raft_consensus.cc:482] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:18.675242 4063 raft_consensus.cc:3037] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:18.678920 4063 raft_consensus.cc:504] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.679893 4063 leader_election.cc:288] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf94d508f4971b950391f4e95b9e3 (127.2.223.129:37417), 2bee4e0b841a4abe98743cf59033573b (127.2.223.131:37239)
I20260807 11:16:18.680310 3705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce5935820ff4aab98b1001d6c19e2c6" candidate_uuid: "311e96806e4043d7a324a38e2fc7839f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf94d508f4971b950391f4e95b9e3"
I20260807 11:16:18.680599 3705 raft_consensus.cc:3037] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:18.680641 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce5935820ff4aab98b1001d6c19e2c6" candidate_uuid: "311e96806e4043d7a324a38e2fc7839f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bee4e0b841a4abe98743cf59033573b"
I20260807 11:16:18.680919 3990 raft_consensus.cc:3037] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:18.684298 3705 raft_consensus.cc:2454] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311e96806e4043d7a324a38e2fc7839f in term 1.
I20260807 11:16:18.684329 3990 raft_consensus.cc:2454] T 3ce5935820ff4aab98b1001d6c19e2c6 P 2bee4e0b841a4abe98743cf59033573b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311e96806e4043d7a324a38e2fc7839f in term 1.
I20260807 11:16:18.684919 3779 leader_election.cc:302] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311e96806e4043d7a324a38e2fc7839f, 74fcf94d508f4971b950391f4e95b9e3; no voters:
I20260807 11:16:18.685410 4063 raft_consensus.cc:2781] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:18.686303 4063 raft_consensus.cc:686] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [term 1 LEADER]: Becoming Leader. State: Replica: 311e96806e4043d7a324a38e2fc7839f, State: Running, Role: LEADER
I20260807 11:16:18.687062 4063 consensus_queue.cc:237] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } }
I20260807 11:16:18.693017 3554 catalog_manager.cc:5495] T 3ce5935820ff4aab98b1001d6c19e2c6 P 311e96806e4043d7a324a38e2fc7839f reported cstate change: term changed from 0 to 1, leader changed from <none> to 311e96806e4043d7a324a38e2fc7839f (127.2.223.130). New cstate: current_term: 1 leader_uuid: "311e96806e4043d7a324a38e2fc7839f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf94d508f4971b950391f4e95b9e3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bee4e0b841a4abe98743cf59033573b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311e96806e4043d7a324a38e2fc7839f" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36713 } health_report { overall_health: HEALTHY } } }
W20260807 11:16:18.744043 3755 tablet.cc:2369] T 3ce5935820ff4aab98b1001d6c19e2c6 P 74fcf94d508f4971b950391f4e95b9e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:18.758381 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.737832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.2.223.129:37417)
0807 11:16:18.738023 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.738028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.738984 (+ 956us) server_negotiation.cc:407] Connection header received
0807 11:16:18.739154 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.739157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.739249 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.739333 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.741710 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.743057 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.745149 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.745788 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.745885 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.751395 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.751408 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.751424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.751460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.755319 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.756147 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.756155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.756157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.756353 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.756834 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.756838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.756840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.757264 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.757533 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.757844 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.758009 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:18.815651 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.797821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.2.223.129:37417)
0807 11:16:18.798094 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.798101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.798325 (+ 224us) server_negotiation.cc:407] Connection header received
0807 11:16:18.798627 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.798632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.798703 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.798827 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.800680 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.801558 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.803242 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.803732 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.803920 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.809352 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.809357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.809361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.809394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.812676 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.813362 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.813366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.813369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.813507 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.814027 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.814050 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.814052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.814392 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.814765 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.815017 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.815247 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:16:18 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:18.879551 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.863455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.2.223.129:37417)
0807 11:16:18.863749 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.863759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.864135 (+ 376us) server_negotiation.cc:407] Connection header received
0807 11:16:18.864368 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.864374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.864458 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.864605 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.866099 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.866998 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.868403 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.868892 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.869047 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.873399 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.873404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.873407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.873443 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.876618 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.877357 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.877364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.877368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.877503 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.878007 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.878013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.878017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.878334 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.878634 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.878942 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.879183 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260807 11:16:18.881963 3664 tablet_service.cc:2908] Rejecting scan request for tablet 3ce5935820ff4aab98b1001d6c19e2c6: Uninitialized: clean time has not yet been initialized
I20260807 11:16:18.886201 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3618
I20260807 11:16:18.900746 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:18.887245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.2.223.129:37417)
0807 11:16:18.887406 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:18.887410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:18.887884 (+ 474us) server_negotiation.cc:407] Connection header received
0807 11:16:18.888018 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:18.888021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:18.888096 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:18.888189 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:18.889562 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.890229 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.891509 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:18.891910 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:18.892015 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:18.894262 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:18.894268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:18.894273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:18.894308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:18.897597 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.898472 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.898500 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.898504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.898660 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:18.899092 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:18.899098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:18.899103 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:18.899635 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:18.899964 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:18.900188 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:16:18.900419 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 11:16:18.901490 3747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:18.991842 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3618
I20260807 11:16:19.014544 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3760
Aug 07 11:16:19 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:19.031756 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:19.015343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.2.223.130:36713)
0807 11:16:19.015810 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:19.015814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:19.015831 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:16:19.015887 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:19.015891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:19.015952 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:19.016044 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:19.017458 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:19.018192 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:19.020238 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:19.020652 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:19.020746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:19.025857 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:19.025869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:19.025886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:19.025921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:19.028929 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:19.029631 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:19.029637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:19.029640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:19.029817 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:19.030275 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:19.030278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:19.030280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:19.030669 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:19.030911 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:19.031174 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 11:16:19.031340 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260807 11:16:19.032503 3892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:19.123819 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3760
I20260807 11:16:19.146978 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3903
Aug 07 11:16:19 dist-test-slave-5l3k krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:19.161908 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:19.147789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.2.223.131:37239)
0807 11:16:19.147968 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:19.147973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:19.147992 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:16:19.148180 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:19.148184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:19.148246 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:19.148384 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:19.149736 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:19.150504 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:19.151853 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:19.152235 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:19.152326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:19.156718 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:19.156732 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:19.156749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:19.156787 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:19.159612 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:19.160231 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:19.160239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:19.160241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:19.160365 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:19.160683 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:19.160686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:19.160688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:19.161014 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:19.161227 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:19.161482 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 11:16:19.161639 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:16:19.162578 4033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:19.250367 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3903
I20260807 11:16:19.269747 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 3522
I20260807 11:16:19.271077 3586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:19.382391 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 3522
2026-08-07T11:16:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4504 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:19 dist-test-slave-5l3k krb5kdc[4091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:19 dist-test-slave-5l3k krb5kdc[4091](info): set up 2 sockets
Aug 07 11:16:19 dist-test-slave-5l3k krb5kdc[4091](info): commencing operation
krb5kdc: starting...
W20260807 11:16:20.530867 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:20 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:20.728305 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41321
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:21.052465 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:21.052814 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:21.052970 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:21.060213 4104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:21.060365 4104 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:16:21.060467 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:21.060593 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:21.060683 4104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:21.060774 4104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:21.070647 4104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:21.071868 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:21.073169 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:21.085203 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:21.085232 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:21.085170 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:21.086275 4104 server_base.cc:1034] running on GCE node
I20260807 11:16:21.087095 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:21.088354 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:21.089572 4104 hybrid_clock.cc:648] HybridClock initialized: now 1786101381089516 us; error 27 us; skew 500 ppm
I20260807 11:16:21.089912 4104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:21 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:21.097227 4104 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:21.098675 4104 webserver.cc:460] Webserver started at http://127.2.223.190:40375/ using document root <none> and password file <none>
I20260807 11:16:21.099372 4104 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:21.099505 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:21.099839 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:21.102841 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "515be0e433544537811c4abf9877f564"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "a811069bc277c8d6826cdc73ad5d4863"
server_key_iv: "54c7f523e76a550f3bdf0d55ecef0f3d"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.103634 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "515be0e433544537811c4abf9877f564"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "a811069bc277c8d6826cdc73ad5d4863"
server_key_iv: "54c7f523e76a550f3bdf0d55ecef0f3d"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.108812 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260807 11:16:21.112143 4119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:21.113354 4104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:16:21.113533 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "515be0e433544537811c4abf9877f564"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "a811069bc277c8d6826cdc73ad5d4863"
server_key_iv: "54c7f523e76a550f3bdf0d55ecef0f3d"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.113754 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:21.140234 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:21.144272 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:21.144585 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:21.167119 4104 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41321
I20260807 11:16:21.167209 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41321 every 8 connection(s)
I20260807 11:16:21.168484 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:21.172442 4184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:21.175891 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4104
I20260807 11:16:21.176126 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:21.176784 2942 external_mini_cluster.cc:1428] Setting key 823b2cb1e85de2fca846f65987776249
I20260807 11:16:21.185945 4184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Bootstrap starting.
I20260807 11:16:21.189568 4184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:21.190610 4184 log.cc:824] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:16:21 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:21.193615 4184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: No bootstrap required, opened a new log
I20260807 11:16:21.200270 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:21.178967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.2.223.190:41321)
0807 11:16:21.179771 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:21.179787 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:21.179841 (+ 54us) server_negotiation.cc:407] Connection header received
0807 11:16:21.181637 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:21.181670 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:21.182208 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:21.182680 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:21.183921 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.185912 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.187321 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.188434 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.188619 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:21.193000 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:21.193008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:21.193022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:21.193061 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:21.196188 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.196960 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.196966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.196971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.197134 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.197521 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.197524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.197526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.198059 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:21.198313 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:21.198842 (+ 529us) server_negotiation.cc:299] Negotiation successful
0807 11:16:21.199323 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":344,"threads_started":1}
I20260807 11:16:21.203143 4184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } }
I20260807 11:16:21.203572 4184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:21.203691 4184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be0e433544537811c4abf9877f564, State: Initialized, Role: FOLLOWER
I20260807 11:16:21.204239 4184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } }
I20260807 11:16:21.204499 4184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:21.204648 4184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:21.204821 4184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:21.207172 4184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } }
I20260807 11:16:21.207641 4184 leader_election.cc:302] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515be0e433544537811c4abf9877f564; no voters:
I20260807 11:16:21.208179 4184 leader_election.cc:288] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:21.208338 4189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:21.209260 4189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [term 1 LEADER]: Becoming Leader. State: Replica: 515be0e433544537811c4abf9877f564, State: Running, Role: LEADER
I20260807 11:16:21.209826 4189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } }
I20260807 11:16:21.210341 4184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:21.213125 4191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515be0e433544537811c4abf9877f564. Latest consensus state: current_term: 1 leader_uuid: "515be0e433544537811c4abf9877f564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } } }
I20260807 11:16:21.213558 4191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:21.214373 4190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515be0e433544537811c4abf9877f564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515be0e433544537811c4abf9877f564" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41321 } } }
I20260807 11:16:21.214854 4190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:21.217300 4207 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:16:21.217518 4198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:21.221590 4198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:21.229866 4198 catalog_manager.cc:1347] Generated new cluster ID: bf8aa84132b24c60984d3bc9881882d2
I20260807 11:16:21.229974 4198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:21.265688 4198 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:21.267742 4198 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:21.275766 4198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Generated new TSK 0
I20260807 11:16:21.276644 4198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:21.334121 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41321
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:21.663686 4212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:21.663955 4212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:21.664127 4212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:21.671253 4212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:21.671394 4212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:21.671562 4212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:21.671693 4212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:21.682384 4212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:21.683846 4212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:21.685398 4212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:21.700780 4222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:21.700749 4220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:21.701014 4219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:21.701987 4212 server_base.cc:1034] running on GCE node
I20260807 11:16:21.702710 4212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:21.703657 4212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:21.704912 4212 hybrid_clock.cc:648] HybridClock initialized: now 1786101381704848 us; error 39 us; skew 500 ppm
I20260807 11:16:21.705216 4212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:21 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:21.708422 4212 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:21.709937 4212 webserver.cc:460] Webserver started at http://127.2.223.129:46223/ using document root <none> and password file <none>
I20260807 11:16:21.710774 4212 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:21.710887 4212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:21.711239 4212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:21.714406 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "f5ad8ea741754c708550d8baeded1728"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "316987b10b96d7841c471ddb71b65687"
server_key_iv: "d55ce919b43a04e1c5e3a4033218ecea"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.715301 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "f5ad8ea741754c708550d8baeded1728"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "316987b10b96d7841c471ddb71b65687"
server_key_iv: "d55ce919b43a04e1c5e3a4033218ecea"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.720613 4212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:21.723980 4229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:21.725242 4212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:16:21.725410 4212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "f5ad8ea741754c708550d8baeded1728"
format_stamp: "Formatted at 2026-08-07 11:16:21 on dist-test-slave-5l3k"
server_key: "316987b10b96d7841c471ddb71b65687"
server_key_iv: "d55ce919b43a04e1c5e3a4033218ecea"
server_key_version: "encryptionkey@0"
I20260807 11:16:21.725621 4212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:21.745137 4212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:21.749574 4212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:21.749897 4212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:21.753151 4212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:21.753273 4212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:21.753474 4212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:21.753602 4212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:21 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:21.778720 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:21.753528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:57275 (local address 127.2.223.190:41321)
0807 11:16:21.754095 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:21.754102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:21.755371 (+ 1269us) server_negotiation.cc:407] Connection header received
0807 11:16:21.757478 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:21.757484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:21.757576 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:21.757699 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:21.759772 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.760816 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.763507 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.764430 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.764621 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:21.770065 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:21.770071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:21.770074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:21.770115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:21.774270 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.775958 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.775966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.775973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.776155 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.776626 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.776635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.776641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.777142 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:21.777342 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:21.778064 (+ 722us) server_negotiation.cc:299] Negotiation successful
0807 11:16:21.778337 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
I20260807 11:16:21.780421 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:21.754342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.129:57275)
0807 11:16:21.755190 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:21.755229 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:21.757192 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:21.757962 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:21.757970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:21.758719 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:21.759582 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:21.759610 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.761020 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.761025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:21.763315 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:21.763336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.764754 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.764763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:21.764968 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:21.765702 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:21.765761 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:21.769792 (+ 4031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:21.774505 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:21.774520 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:21.774539 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:21.775374 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:21.776287 (+ 913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:21.776290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:21.776293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:21.776488 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:21.777279 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:21.777284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:21.777612 (+ 328us) client_negotiation.cc:769] Sending connection context
0807 11:16:21.777920 (+ 308us) client_negotiation.cc:240] Negotiation successful
0807 11:16:21.778463 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":348,"threads_started":1}
I20260807 11:16:21.804194 4212 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:46067
I20260807 11:16:21.804234 4354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:46067 every 8 connection(s)
I20260807 11:16:21.804961 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:21.788144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.129:56517)
0807 11:16:21.788837 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:21.788862 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:21.789026 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:21.789472 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:21.789478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:21.790254 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:21.790836 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:21.790865 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.792607 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.792613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:21.793660 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:21.793672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.794840 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.794845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:21.794963 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:21.795621 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:21.795653 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:21.797009 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:21.801701 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:21.801712 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:21.801718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:21.802258 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:21.802743 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:21.802750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:21.802756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:21.802951 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:21.803757 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:21.803765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:21.804031 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 11:16:21.804240 (+ 209us) client_negotiation.cc:240] Negotiation successful
0807 11:16:21.804637 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":382,"threads_started":1}
I20260807 11:16:21.805023 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:21.787986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:56517 (local address 127.2.223.190:41321)
0807 11:16:21.788210 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:21.788219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:21.788969 (+ 750us) server_negotiation.cc:407] Connection header received
0807 11:16:21.789183 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:21.789190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:21.789295 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:21.789424 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:21.791057 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.792441 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.793853 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:21.794650 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:21.794804 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:21.797256 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:21.797266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:21.797273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:21.797317 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:21.801554 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.802392 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.802400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.802405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.802604 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:21.803098 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:21.803108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:21.803114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:21.803590 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:21.803769 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:21.804404 (+ 635us) server_negotiation.cc:299] Negotiation successful
0807 11:16:21.804661 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:16:21.806053 4212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:21.810539 4355 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41321
I20260807 11:16:21.811064 4355 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:21.812542 4355 heartbeater.cc:511] Master 127.2.223.190:41321 requested a full tablet report, sending...
I20260807 11:16:21.814893 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4212
I20260807 11:16:21.815044 4137 ts_manager.cc:194] Registered new tserver with Master: f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067)
I20260807 11:16:21.815158 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:21.815817 2942 external_mini_cluster.cc:1428] Setting key 1b43ad9b21bcfdae366d37f15b9c7cad
I20260807 11:16:21.816411 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:57275
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:21.866891 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41321
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:22.201297 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:22.201629 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:22.201802 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:22.209051 4360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:22.209195 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:22.209349 4360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:22.209491 4360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:22.219750 4360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:22.221101 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:22.222523 4360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:22.236078 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:22.236035 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:22.235981 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:22.236994 4360 server_base.cc:1034] running on GCE node
I20260807 11:16:22.237601 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:22.238518 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:22.239781 4360 hybrid_clock.cc:648] HybridClock initialized: now 1786101382239712 us; error 40 us; skew 500 ppm
I20260807 11:16:22.240082 4360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:22 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:22.243297 4360 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:22.244863 4360 webserver.cc:460] Webserver started at http://127.2.223.130:44909/ using document root <none> and password file <none>
I20260807 11:16:22.245609 4360 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:22.245743 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:22.246073 4360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:22.249074 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "782b04af5c054992b6a58a7916e73469"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "3b5a0f9bf43b99bc4f51209553c38229"
server_key_iv: "0387e4882469d6158aa1ed8b791430d1"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.249891 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "782b04af5c054992b6a58a7916e73469"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "3b5a0f9bf43b99bc4f51209553c38229"
server_key_iv: "0387e4882469d6158aa1ed8b791430d1"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.255010 4360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:16:22.258301 4375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:22.259629 4360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:22.259810 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "782b04af5c054992b6a58a7916e73469"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "3b5a0f9bf43b99bc4f51209553c38229"
server_key_iv: "0387e4882469d6158aa1ed8b791430d1"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.260042 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:22.316073 4360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:22.320393 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:22.320675 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:22.323545 4360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:22.323678 4360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:22.323860 4360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:22.323940 4360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:22 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:22.347574 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.324021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:51095 (local address 127.2.223.190:41321)
0807 11:16:22.324485 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:22.324494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:22.325722 (+ 1228us) server_negotiation.cc:407] Connection header received
0807 11:16:22.327981 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:22.327990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:22.328118 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:22.328254 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:22.330651 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.331731 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.334437 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.335204 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.335368 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.339920 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:22.339930 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:22.339937 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:22.339995 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:22.344143 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.344875 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.344882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.344889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.345055 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.345630 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.345639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.345642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.345901 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:22.346058 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:22.346982 (+ 924us) server_negotiation.cc:299] Negotiation successful
0807 11:16:22.347197 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":238,"threads_started":1}
I20260807 11:16:22.348621 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.324766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.130:51095)
0807 11:16:22.325548 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:22.325605 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:22.327606 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:22.328488 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:22.328501 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:22.329401 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:22.330394 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.330414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.331889 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.331895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.334262 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.334282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.335497 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.335507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.335823 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.336607 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:22.336659 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:22.339669 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:22.344293 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.344307 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.344322 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.344739 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.345209 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.345215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.345222 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.345493 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.346033 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:22.346045 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:22.346584 (+ 539us) client_negotiation.cc:769] Sending connection context
0807 11:16:22.346875 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 11:16:22.347401 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":364,"threads_started":1}
I20260807 11:16:22.368238 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.354518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:54541 (local address 127.2.223.190:41321)
0807 11:16:22.354751 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:22.354759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:22.355358 (+ 599us) server_negotiation.cc:407] Connection header received
0807 11:16:22.355551 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:22.355556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:22.355631 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:22.355729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:22.356649 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.357358 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.358713 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.359160 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.359295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.361597 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:22.361606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:22.361614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:22.361664 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:22.365052 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.366014 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.366018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.366020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.366154 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.366628 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.366635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.366640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.367042 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:22.367171 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:22.367815 (+ 644us) server_negotiation.cc:299] Negotiation successful
0807 11:16:22.368000 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:16:22.368839 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.354631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.130:54541)
0807 11:16:22.355249 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:22.355264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:22.355438 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:22.355756 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:22.355761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:22.356167 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:22.356464 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.356473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.357498 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.357505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.358579 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.358591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.359373 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.359381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.359564 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.360253 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:22.360268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:22.361342 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:22.365192 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.365200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.365207 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.365887 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.366269 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.366276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.366282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.366504 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.367183 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:22.367190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:22.367483 (+ 293us) client_negotiation.cc:769] Sending connection context
0807 11:16:22.367659 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 11:16:22.368146 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":361,"threads_started":1}
I20260807 11:16:22.385632 4360 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:34821
I20260807 11:16:22.385670 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:34821 every 8 connection(s)
I20260807 11:16:22.386945 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:22.387437 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4360
I20260807 11:16:22.387663 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:22.388110 2942 external_mini_cluster.cc:1428] Setting key 117025b1de11b396657b0abf79e9a803
I20260807 11:16:22.389569 4503 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41321
I20260807 11:16:22.389859 4503 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:22.391392 4503 heartbeater.cc:511] Master 127.2.223.190:41321 requested a full tablet report, sending...
I20260807 11:16:22.393170 4137 ts_manager.cc:194] Registered new tserver with Master: 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821)
I20260807 11:16:22.394433 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:51095
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:22.437378 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41321
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:22.813374 4509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:22.813694 4509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:22.813879 4509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 11:16:22.819834 4355 heartbeater.cc:503] Master 127.2.223.190:41321 was elected leader, sending a full tablet report...
W20260807 11:16:22.821574 4509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:22.821689 4509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:22.821859 4509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:22.821976 4509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:22.832089 4509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:22.833325 4509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:22.834759 4509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:22.848047 4515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:22.849050 4509 server_base.cc:1034] running on GCE node
W20260807 11:16:22.851902 4517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:22.851974 4514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:22.853039 4509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:22.854440 4509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:22.855763 4509 hybrid_clock.cc:648] HybridClock initialized: now 1786101382855673 us; error 52 us; skew 500 ppm
I20260807 11:16:22.856197 4509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:22 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:22.859826 4509 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:22.861248 4509 webserver.cc:460] Webserver started at http://127.2.223.131:44329/ using document root <none> and password file <none>
I20260807 11:16:22.861928 4509 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:22.862015 4509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:22.862270 4509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:22.865551 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "4b4b3500f5e74bb996a7cf42208aefaf"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "cbe362f043d7920756ed0783c55b04bd"
server_key_iv: "94cf96ba0b8ec477df3a916819573c43"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.866313 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "4b4b3500f5e74bb996a7cf42208aefaf"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "cbe362f043d7920756ed0783c55b04bd"
server_key_iv: "94cf96ba0b8ec477df3a916819573c43"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.871732 4509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:16:22.875245 4525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:22.876665 4509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:22.876852 4509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "4b4b3500f5e74bb996a7cf42208aefaf"
format_stamp: "Formatted at 2026-08-07 11:16:22 on dist-test-slave-5l3k"
server_key: "cbe362f043d7920756ed0783c55b04bd"
server_key_iv: "94cf96ba0b8ec477df3a916819573c43"
server_key_version: "encryptionkey@0"
I20260807 11:16:22.877038 4509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:22.888330 4509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:22.892202 4509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:22.892482 4509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:22.895061 4509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:22.895167 4509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:22.895320 4509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:22.895416 4509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:22 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:22.919240 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.895991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:47371 (local address 127.2.223.190:41321)
0807 11:16:22.896494 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:22.896501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:22.897565 (+ 1064us) server_negotiation.cc:407] Connection header received
0807 11:16:22.899939 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:22.899946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:22.900044 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:22.900177 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:22.902348 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.903525 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.906185 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.906713 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.906855 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.911788 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:22.911793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:22.911797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:22.911831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:22.915098 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.916162 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.916166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.916168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.916286 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.916884 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.916892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.916898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.917345 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:22.917516 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:22.918362 (+ 846us) server_negotiation.cc:299] Negotiation successful
0807 11:16:22.918667 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260807 11:16:22.920023 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.896439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.131:47371)
0807 11:16:22.897362 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:22.897421 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:22.899626 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:22.900415 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:22.900422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:22.901131 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:22.902136 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.902164 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.903720 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.903728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.905988 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.906014 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.906971 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.906980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.907400 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.908253 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:22.908294 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:22.911557 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:22.915298 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.915309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.915351 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.916020 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.916413 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.916417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.916419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.916761 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.917524 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:22.917533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:22.917952 (+ 419us) client_negotiation.cc:769] Sending connection context
0807 11:16:22.918190 (+ 238us) client_negotiation.cc:240] Negotiation successful
0807 11:16:22.918708 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":416,"threads_started":1}
I20260807 11:16:22.941888 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.926407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.2.223.131:33073)
0807 11:16:22.927285 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:22.927314 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:22.927497 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:22.927960 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:22.927964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:22.928639 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:22.929190 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.929201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.930678 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.930682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.932048 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:22.932060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.932966 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.932977 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:22.933289 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.933972 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:22.934002 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:22.935466 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:22.939146 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.939151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.939155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.939781 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.940175 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:22.940180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:22.940184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:22.940331 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:22.940857 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:22.940865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:22.941061 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 11:16:22.941251 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 11:16:22.941562 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":513,"threads_started":1}
I20260807 11:16:22.942193 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.926186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:33073 (local address 127.2.223.190:41321)
0807 11:16:22.926408 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:22.926416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:22.927443 (+ 1027us) server_negotiation.cc:407] Connection header received
0807 11:16:22.927665 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:22.927699 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:22.927810 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:22.927944 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:22.929403 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.930424 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.932207 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:22.932691 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:22.932827 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:22.935709 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:22.935716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:22.935724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:22.935768 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:22.939024 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.939918 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.939924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.939927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.940059 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:22.940447 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:22.940452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:22.940455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:22.940735 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:22.940900 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:22.941394 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 11:16:22.941685 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:16:22.953274 4509 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:36003
I20260807 11:16:22.953301 4651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:36003 every 8 connection(s)
I20260807 11:16:22.954623 4509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:22.956664 4652 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41321
I20260807 11:16:22.956883 4652 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:22.957499 4652 heartbeater.cc:511] Master 127.2.223.190:41321 requested a full tablet report, sending...
I20260807 11:16:22.958626 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4509
I20260807 11:16:22.958845 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:22.959003 4137 ts_manager.cc:194] Registered new tserver with Master: 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131:36003)
I20260807 11:16:22.959383 2942 external_mini_cluster.cc:1428] Setting key e1c948da69fdb82d7cc72da9ef712e97
I20260807 11:16:22.959951 4137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:47371
I20260807 11:16:22.962296 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:16:22 dist-test-slave-5l3k krb5kdc[4091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:16:23 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:23.013180 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:22.999033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.2.223.190:41321)
0807 11:16:22.999300 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:22.999310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:22.999552 (+ 242us) server_negotiation.cc:407] Connection header received
0807 11:16:22.999791 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:22.999795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:22.999863 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:22.999976 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:23.001324 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.002069 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.003468 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.003900 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.004039 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.007899 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:23.007904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:23.007907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:23.007943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:23.010420 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:23.011172 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:23.011177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:23.011181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:23.011317 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:23.011699 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:23.011705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:23.011708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:23.012048 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:23.012323 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.012610 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.012829 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 11:16:23.017655 4137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:23.024680 4137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:23.063007 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.036600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:46067 (local address 127.0.0.1:39790)
0807 11:16:23.053084 (+ 16484us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.053108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.053277 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.053919 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.053930 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.053974 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.054724 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.054742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.057249 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.057258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.059803 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.059820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.061857 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.061863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.062116 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.062174 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.062323 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.062531 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16291,"mutex_wait_us":27,"thread_start_us":12210,"threads_started":1}
I20260807 11:16:23.063128 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.037431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.2.223.129:46067)
0807 11:16:23.050479 (+ 13048us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.050489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.053232 (+ 2743us) server_negotiation.cc:407] Connection header received
0807 11:16:23.053455 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.053462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.053670 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.053867 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.054972 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.057026 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.060003 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.061637 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.061787 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.062016 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.062651 (+ 635us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.062823 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12602,"thread_start_us":12754,"threads_started":1}
I20260807 11:16:23.063824 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.036595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.2.223.130:34821)
0807 11:16:23.053068 (+ 16473us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.053075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.053099 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:16:23.053212 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.053220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.053491 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.053700 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.056092 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.057858 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.060494 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.062044 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.062195 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.062401 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.063328 (+ 927us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.063504 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16321,"thread_start_us":17450,"threads_started":1}
I20260807 11:16:23.063143 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.036401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:34821 (local address 127.0.0.1:46700)
0807 11:16:23.049299 (+ 12898us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.049360 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.049562 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.053946 (+ 4384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.053952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.054003 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.055842 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.055862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.058024 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.058035 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.060331 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.060345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.062283 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.062292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.062453 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.062510 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.062593 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.062728 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12541,"mutex_wait_us":3,"spinlock_wait_cycles":8064,"thread_start_us":16968,"threads_started":1}
I20260807 11:16:23.066305 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.054379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:36003 (local address 127.0.0.1:53714)
0807 11:16:23.058525 (+ 4146us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.058551 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.058706 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.059221 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.059226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.059249 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.059655 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.059664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.061473 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.061478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.063720 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.063737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.065520 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.065527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.065719 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.065753 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.065862 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.065982 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3999,"thread_start_us":3815,"threads_started":1}
I20260807 11:16:23.066427 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.050292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.2.223.131:36003)
0807 11:16:23.057737 (+ 7445us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.057746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.058655 (+ 909us) server_negotiation.cc:407] Connection header received
0807 11:16:23.058875 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.058880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.059034 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.059223 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.059858 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.061296 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.063945 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.065306 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.065452 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.065678 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.065915 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.066116 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7275,"thread_start_us":7101,"threads_started":1}
I20260807 11:16:23.067670 4428 tablet_service.cc:1467] Processing CreateTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 (DEFAULT_TABLE table=test-table [id=283db014c05e40a29c978fe6000dbf24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.067579 4282 tablet_service.cc:1467] Processing CreateTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 (DEFAULT_TABLE table=test-table [id=283db014c05e40a29c978fe6000dbf24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.068862 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06c3c89ca65492e9d75fd1020f38e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.068981 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06c3c89ca65492e9d75fd1020f38e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.069839 4578 tablet_service.cc:1467] Processing CreateTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 (DEFAULT_TABLE table=test-table [id=283db014c05e40a29c978fe6000dbf24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.071218 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06c3c89ca65492e9d75fd1020f38e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.080754 4674 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Bootstrap starting.
I20260807 11:16:23.084023 4675 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Bootstrap starting.
I20260807 11:16:23.085471 4676 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Bootstrap starting.
I20260807 11:16:23.085697 4674 tablet_bootstrap.cc:654] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.087075 4674 log.cc:824] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:23.087813 4675 tablet_bootstrap.cc:654] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.088506 4676 tablet_bootstrap.cc:654] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.088958 4675 log.cc:824] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:23.089552 4676 log.cc:824] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:23.090308 4674 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: No bootstrap required, opened a new log
I20260807 11:16:23.090631 4674 ts_tablet_manager.cc:1397] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260807 11:16:23.092089 4675 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: No bootstrap required, opened a new log
I20260807 11:16:23.092355 4675 ts_tablet_manager.cc:1397] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:16:23.092489 4676 tablet_bootstrap.cc:492] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: No bootstrap required, opened a new log
I20260807 11:16:23.092730 4676 ts_tablet_manager.cc:1397] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:16:23.098907 4674 raft_consensus.cc:348] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.099287 4674 raft_consensus.cc:374] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.099375 4674 raft_consensus.cc:729] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782b04af5c054992b6a58a7916e73469, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.099943 4674 consensus_queue.cc:260] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.100998 4503 heartbeater.cc:503] Master 127.2.223.190:41321 was elected leader, sending a full tablet report...
I20260807 11:16:23.101645 4674 ts_tablet_manager.cc:1428] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260807 11:16:23.103127 4676 raft_consensus.cc:348] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.103052 4675 raft_consensus.cc:348] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.103502 4676 raft_consensus.cc:374] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.103502 4675 raft_consensus.cc:374] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.103622 4675 raft_consensus.cc:729] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad8ea741754c708550d8baeded1728, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.103622 4676 raft_consensus.cc:729] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4b3500f5e74bb996a7cf42208aefaf, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.104190 4676 consensus_queue.cc:260] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.104190 4675 consensus_queue.cc:260] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.105458 4652 heartbeater.cc:503] Master 127.2.223.190:41321 was elected leader, sending a full tablet report...
I20260807 11:16:23.106361 4676 ts_tablet_manager.cc:1428] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 11:16:23.108587 4675 ts_tablet_manager.cc:1428] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
W20260807 11:16:23.144937 4504 tablet.cc:2369] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:23.162803 4682 raft_consensus.cc:482] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:23.163197 4682 raft_consensus.cc:504] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.165191 4682 leader_election.cc:288] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821), 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131:36003)
I20260807 11:16:23.173064 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.165634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38703 (local address 127.2.223.131:36003)
0807 11:16:23.165894 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.165903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.167211 (+ 1308us) server_negotiation.cc:407] Connection header received
0807 11:16:23.167476 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.167481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.167562 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.167630 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.168400 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.169120 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.171161 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.172104 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.172274 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.172346 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.172674 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.172806 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:16:23.173116 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.165537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:36003 (local address 127.2.223.129:38703)
0807 11:16:23.167115 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.167132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.167323 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.167844 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.167848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.167872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.168251 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.168260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.169263 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.169269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.170980 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.170992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.172319 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.172325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.172484 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.172516 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.172614 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.172735 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":61,"thread_start_us":478,"threads_started":1}
I20260807 11:16:23.174345 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.165579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:57203 (local address 127.2.223.130:34821)
0807 11:16:23.165801 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.165810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.166830 (+ 1020us) server_negotiation.cc:407] Connection header received
0807 11:16:23.167073 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.167080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.167173 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.167264 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.168052 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.169195 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.171746 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.172833 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.172984 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.173051 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.173975 (+ 924us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.174109 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:16:23.174217 4606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06c3c89ca65492e9d75fd1020f38e51" candidate_uuid: "f5ad8ea741754c708550d8baeded1728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" is_pre_election: true
I20260807 11:16:23.174660 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.165573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:34821 (local address 127.2.223.129:57203)
0807 11:16:23.166719 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.166740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.166898 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.167446 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.167450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.167473 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.167847 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.167857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.169367 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.169373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.171572 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.171588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.173608 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.173612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.173785 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.173826 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.173933 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.174055 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":833,"threads_started":1}
I20260807 11:16:23.174858 4606 raft_consensus.cc:2454] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ad8ea741754c708550d8baeded1728 in term 0.
I20260807 11:16:23.175557 4232 leader_election.cc:302] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b3500f5e74bb996a7cf42208aefaf, f5ad8ea741754c708550d8baeded1728; no voters:
I20260807 11:16:23.175726 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06c3c89ca65492e9d75fd1020f38e51" candidate_uuid: "f5ad8ea741754c708550d8baeded1728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469" is_pre_election: true
I20260807 11:16:23.175953 4682 raft_consensus.cc:2781] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:23.176116 4682 raft_consensus.cc:482] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:23.176229 4682 raft_consensus.cc:3037] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.176178 4453 raft_consensus.cc:2454] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ad8ea741754c708550d8baeded1728 in term 0.
I20260807 11:16:23.178624 4682 raft_consensus.cc:504] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.179221 4682 leader_election.cc:288] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [CANDIDATE]: Term 1 election: Requested vote from peers 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821), 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131:36003)
I20260807 11:16:23.179664 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06c3c89ca65492e9d75fd1020f38e51" candidate_uuid: "f5ad8ea741754c708550d8baeded1728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469"
I20260807 11:16:23.179720 4606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06c3c89ca65492e9d75fd1020f38e51" candidate_uuid: "f5ad8ea741754c708550d8baeded1728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4b3500f5e74bb996a7cf42208aefaf"
I20260807 11:16:23.179858 4453 raft_consensus.cc:3037] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.179913 4606 raft_consensus.cc:3037] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.182457 4453 raft_consensus.cc:2454] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ad8ea741754c708550d8baeded1728 in term 1.
I20260807 11:16:23.182915 4230 leader_election.cc:302] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782b04af5c054992b6a58a7916e73469, f5ad8ea741754c708550d8baeded1728; no voters:
I20260807 11:16:23.183207 4606 raft_consensus.cc:2454] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ad8ea741754c708550d8baeded1728 in term 1.
I20260807 11:16:23.183265 4682 raft_consensus.cc:2781] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:23.184007 4682 raft_consensus.cc:686] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 1 LEADER]: Becoming Leader. State: Replica: f5ad8ea741754c708550d8baeded1728, State: Running, Role: LEADER
I20260807 11:16:23.184489 4682 consensus_queue.cc:237] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.188947 4137 catalog_manager.cc:5495] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ad8ea741754c708550d8baeded1728 (127.2.223.129). New cstate: current_term: 1 leader_uuid: "f5ad8ea741754c708550d8baeded1728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } health_report { overall_health: UNKNOWN } } }
W20260807 11:16:23.206187 4653 tablet.cc:2369] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:23.218068 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.212609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.2.223.129:46067)
0807 11:16:23.212810 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.212815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.212939 (+ 124us) server_negotiation.cc:407] Connection header received
0807 11:16:23.213228 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.213232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.213360 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.213478 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:23.213869 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.214672 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.215936 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.216364 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.216456 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.216946 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:23.217501 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:23.217700 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.217754 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.217817 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:16:23.229838 4453 raft_consensus.cc:1261] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Refusing update from remote peer f5ad8ea741754c708550d8baeded1728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:23.231050 4682 consensus_queue.cc:1035] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:23.231649 4606 raft_consensus.cc:1261] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Refusing update from remote peer f5ad8ea741754c708550d8baeded1728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:23.232452 4682 consensus_queue.cc:1035] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:23.258333 4690 mvcc.cc:205] Tried to move back new op lower bound from 7315871265696407552 to 7315871265527320576. Current Snapshot: MvccSnapshot[applied={T|T < 7315871265696407552}]
I20260807 11:16:23.274344 4137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:23.274736 4137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:23.279778 4137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Sending DeleteTablet for 3 replicas of tablet d06c3c89ca65492e9d75fd1020f38e51
I20260807 11:16:23.281296 4282 tablet_service.cc:1514] Processing DeleteTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:23 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:39790
I20260807 11:16:23.281282 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:23 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260807 11:16:23.281915 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet d06c3c89ca65492e9d75fd1020f38e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:23 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:53714
I20260807 11:16:23.283154 4137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:23.284528 4137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:23.293744 4700 tablet_replica.cc:330] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: stopping tablet replica
I20260807 11:16:23.294799 4700 raft_consensus.cc:2229] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:23.295801 4700 raft_consensus.cc:2258] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:23.295964 4281 tablet_service.cc:1467] Processing CreateTablet for tablet f1d801c2916340ea9b680af011dec4c2 (DEFAULT_TABLE table=test-table [id=4a6dd22c04de4228947e5a00ff7d46df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.297106 4281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d801c2916340ea9b680af011dec4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.297454 4427 tablet_service.cc:1467] Processing CreateTablet for tablet f1d801c2916340ea9b680af011dec4c2 (DEFAULT_TABLE table=test-table [id=4a6dd22c04de4228947e5a00ff7d46df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.298983 4701 tablet_replica.cc:330] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: stopping tablet replica
I20260807 11:16:23.299532 4701 raft_consensus.cc:2229] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:23.299612 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d801c2916340ea9b680af011dec4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.300243 4701 raft_consensus.cc:2258] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:23.303584 4701 ts_tablet_manager.cc:1905] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:23.306142 4700 ts_tablet_manager.cc:1905] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:23.307195 4578 tablet_service.cc:1467] Processing CreateTablet for tablet f1d801c2916340ea9b680af011dec4c2 (DEFAULT_TABLE table=test-table [id=4a6dd22c04de4228947e5a00ff7d46df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:23.308035 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d801c2916340ea9b680af011dec4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.312554 4701 ts_tablet_manager.cc:1918] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:23.312906 4701 log.cc:1196] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/d06c3c89ca65492e9d75fd1020f38e51
I20260807 11:16:23.313688 4701 ts_tablet_manager.cc:1939] T d06c3c89ca65492e9d75fd1020f38e51 P 782b04af5c054992b6a58a7916e73469: Deleting consensus metadata
I20260807 11:16:23.314215 4699 tablet_replica.cc:330] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: stopping tablet replica
I20260807 11:16:23.316015 4675 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Bootstrap starting.
I20260807 11:16:23.316164 4699 raft_consensus.cc:2229] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:23.316452 4120 catalog_manager.cc:4841] TS 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821): tablet d06c3c89ca65492e9d75fd1020f38e51 (table test-table [id=283db014c05e40a29c978fe6000dbf24]) successfully deleted
I20260807 11:16:23.317317 4699 raft_consensus.cc:2258] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:23.318969 4675 tablet_bootstrap.cc:654] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.320328 4700 ts_tablet_manager.cc:1918] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:23.320573 4700 log.cc:1196] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/d06c3c89ca65492e9d75fd1020f38e51
I20260807 11:16:23.321362 4700 ts_tablet_manager.cc:1939] T d06c3c89ca65492e9d75fd1020f38e51 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting consensus metadata
I20260807 11:16:23.321647 4699 ts_tablet_manager.cc:1905] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:23.323788 4122 catalog_manager.cc:4841] TS 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131:36003): tablet d06c3c89ca65492e9d75fd1020f38e51 (table test-table [id=283db014c05e40a29c978fe6000dbf24]) successfully deleted
I20260807 11:16:23.325006 4675 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: No bootstrap required, opened a new log
I20260807 11:16:23.325380 4676 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Bootstrap starting.
I20260807 11:16:23.327965 4676 tablet_bootstrap.cc:654] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.330261 4675 ts_tablet_manager.cc:1397] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260807 11:16:23.331314 4699 ts_tablet_manager.cc:1918] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:23.331624 4699 log.cc:1196] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/d06c3c89ca65492e9d75fd1020f38e51
I20260807 11:16:23.332257 4674 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Bootstrap starting.
I20260807 11:16:23.331907 4675 raft_consensus.cc:348] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.332690 4675 raft_consensus.cc:374] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.332873 4675 raft_consensus.cc:729] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad8ea741754c708550d8baeded1728, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.333310 4675 consensus_queue.cc:260] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.334218 4675 ts_tablet_manager.cc:1428] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 11:16:23.334571 4676 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: No bootstrap required, opened a new log
I20260807 11:16:23.334829 4676 ts_tablet_manager.cc:1397] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 11:16:23.335335 4699 ts_tablet_manager.cc:1939] T d06c3c89ca65492e9d75fd1020f38e51 P f5ad8ea741754c708550d8baeded1728: Deleting consensus metadata
I20260807 11:16:23.336120 4676 raft_consensus.cc:348] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.336419 4676 raft_consensus.cc:374] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.336621 4676 raft_consensus.cc:729] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4b3500f5e74bb996a7cf42208aefaf, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.337001 4676 consensus_queue.cc:260] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.338219 4676 ts_tablet_manager.cc:1428] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:23.340303 4122 catalog_manager.cc:4841] TS f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067): tablet d06c3c89ca65492e9d75fd1020f38e51 (table test-table [id=283db014c05e40a29c978fe6000dbf24]) successfully deleted
I20260807 11:16:23.343828 4674 tablet_bootstrap.cc:654] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.347290 4674 tablet_bootstrap.cc:492] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: No bootstrap required, opened a new log
I20260807 11:16:23.347476 4674 ts_tablet_manager.cc:1397] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20260807 11:16:23.348522 4674 raft_consensus.cc:348] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.348809 4674 raft_consensus.cc:374] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.348994 4674 raft_consensus.cc:729] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782b04af5c054992b6a58a7916e73469, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.349515 4674 consensus_queue.cc:260] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.351182 4674 ts_tablet_manager.cc:1428] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260807 11:16:23.564146 4356 tablet.cc:2369] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:23.582018 4681 raft_consensus.cc:482] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:23.582302 4681 raft_consensus.cc:504] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.584353 4681 leader_election.cc:288] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067), 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821)
I20260807 11:16:23.592949 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.585096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:38161 (local address 127.2.223.129:46067)
0807 11:16:23.585324 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.585332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.585675 (+ 343us) server_negotiation.cc:407] Connection header received
0807 11:16:23.586054 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.586063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.586203 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.586321 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.586980 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.588388 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.590531 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.591641 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.591838 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.591924 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.592325 (+ 401us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.592485 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:16:23.593000 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.584655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:46067 (local address 127.2.223.131:38161)
0807 11:16:23.585545 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.585565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.585829 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.586351 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.586355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.586380 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.586685 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.586697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.588566 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.588574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.590303 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.590316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.591819 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.591826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.592135 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.592160 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.592296 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.592441 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":681,"threads_started":1}
I20260807 11:16:23.594373 4302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d801c2916340ea9b680af011dec4c2" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad8ea741754c708550d8baeded1728" is_pre_election: true
I20260807 11:16:23.595206 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.584883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:57633 (local address 127.2.223.130:34821)
0807 11:16:23.585084 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.585091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.586128 (+ 1037us) server_negotiation.cc:407] Connection header received
0807 11:16:23.586247 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.586253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.586352 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.586456 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:23.587418 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.588484 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.592034 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.593636 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.593815 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.593950 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.594562 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.594732 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:16:23.595254 4302 raft_consensus.cc:2454] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 0.
I20260807 11:16:23.595391 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.584683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:34821 (local address 127.2.223.131:57633)
0807 11:16:23.585983 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.586001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.586124 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.586571 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.586580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.586606 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:23.587256 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.587266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.588658 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.588664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.591762 (+ 3098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.591778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.594060 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.594068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.594327 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.594373 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.594514 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.594683 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"spinlock_wait_cycles":2944,"thread_start_us":993,"threads_started":1}
I20260807 11:16:23.596319 4528 leader_election.cc:302] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b3500f5e74bb996a7cf42208aefaf, f5ad8ea741754c708550d8baeded1728; no voters:
I20260807 11:16:23.596582 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d801c2916340ea9b680af011dec4c2" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469" is_pre_election: true
I20260807 11:16:23.596737 4681 raft_consensus.cc:2781] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:23.596874 4453 raft_consensus.cc:2454] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 0.
I20260807 11:16:23.596988 4681 raft_consensus.cc:482] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:23.597146 4681 raft_consensus.cc:3037] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.600102 4681 raft_consensus.cc:504] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.601054 4681 leader_election.cc:288] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 election: Requested vote from peers f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067), 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821)
I20260807 11:16:23.601609 4302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d801c2916340ea9b680af011dec4c2" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad8ea741754c708550d8baeded1728"
I20260807 11:16:23.601773 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d801c2916340ea9b680af011dec4c2" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469"
I20260807 11:16:23.601917 4302 raft_consensus.cc:3037] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.602061 4453 raft_consensus.cc:3037] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.606172 4453 raft_consensus.cc:2454] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 1.
I20260807 11:16:23.606340 4302 raft_consensus.cc:2454] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 1.
I20260807 11:16:23.606931 4526 leader_election.cc:302] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b3500f5e74bb996a7cf42208aefaf, 782b04af5c054992b6a58a7916e73469; no voters:
I20260807 11:16:23.607426 4681 raft_consensus.cc:2781] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:23.608314 4681 raft_consensus.cc:686] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 LEADER]: Becoming Leader. State: Replica: 4b4b3500f5e74bb996a7cf42208aefaf, State: Running, Role: LEADER
I20260807 11:16:23.608884 4681 consensus_queue.cc:237] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.613785 4137 catalog_manager.cc:5495] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131). New cstate: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } health_report { overall_health: HEALTHY } } }
Aug 07 11:16:23 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:23.707168 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.682281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41321 (local address 127.0.0.1:46346)
0807 11:16:23.685112 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:23.685146 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:23.685396 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:23.688153 (+ 2757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:23.688161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:23.689552 (+ 1391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:23.690213 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.690232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.692727 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.692733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.694260 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:23.694295 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.695404 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.695424 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:23.695600 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.696392 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:23.696419 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:23.699309 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:23.703627 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:23.703639 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:23.703645 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:23.704405 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:23.704864 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:23.704867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:23.704870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:23.705112 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:23.705861 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:23.705868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:23.706136 (+ 268us) client_negotiation.cc:769] Sending connection context
0807 11:16:23.706307 (+ 171us) client_negotiation.cc:240] Negotiation successful
0807 11:16:23.706581 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2468,"spinlock_wait_cycles":12544,"thread_start_us":3262,"threads_started":1}
I20260807 11:16:23.707293 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:23.682281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.2.223.190:41321)
0807 11:16:23.687723 (+ 5442us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:23.687729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:23.687766 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:16:23.687841 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:23.687848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:23.687928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:23.688071 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:23.690552 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.692486 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.694439 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:23.695099 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:23.695278 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:23.699524 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:23.699531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:23.699536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:23.699577 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:23.703476 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:23.704527 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:23.704535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:23.704538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:23.704720 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:23.705246 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:23.705252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:23.705255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:23.705708 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:23.705861 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:23.706466 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 11:16:23.706776 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":5251,"thread_start_us":4983,"threads_started":1}
I20260807 11:16:23.714139 4137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:46346:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:16:23.715402 4137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:23.722518 4427 tablet_service.cc:1467] Processing CreateTablet for tablet a44a36113a9d49bca600b61f8c9b9ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a0386c8b43d4653bbfa1091500f9b6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:23.722800 4578 tablet_service.cc:1467] Processing CreateTablet for tablet a44a36113a9d49bca600b61f8c9b9ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a0386c8b43d4653bbfa1091500f9b6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:23.723086 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44a36113a9d49bca600b61f8c9b9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.723511 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44a36113a9d49bca600b61f8c9b9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.726827 4281 tablet_service.cc:1467] Processing CreateTablet for tablet a44a36113a9d49bca600b61f8c9b9ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a0386c8b43d4653bbfa1091500f9b6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:23.727368 4281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44a36113a9d49bca600b61f8c9b9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:23.729539 4674 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: Bootstrap starting.
I20260807 11:16:23.731477 4674 tablet_bootstrap.cc:654] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.731747 4676 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: Bootstrap starting.
I20260807 11:16:23.734411 4676 tablet_bootstrap.cc:654] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.735195 4674 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: No bootstrap required, opened a new log
I20260807 11:16:23.735402 4674 ts_tablet_manager.cc:1397] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260807 11:16:23.736387 4674 raft_consensus.cc:348] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.736804 4674 raft_consensus.cc:374] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.736987 4674 raft_consensus.cc:729] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782b04af5c054992b6a58a7916e73469, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.737426 4675 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: Bootstrap starting.
I20260807 11:16:23.737463 4674 consensus_queue.cc:260] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.739058 4680 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:23.739650 4680 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:23.740482 4675 tablet_bootstrap.cc:654] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:23.741259 4674 ts_tablet_manager.cc:1428] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260807 11:16:23.743395 4676 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: No bootstrap required, opened a new log
I20260807 11:16:23.743712 4676 ts_tablet_manager.cc:1397] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260807 11:16:23.744716 4676 raft_consensus.cc:348] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.745038 4676 raft_consensus.cc:374] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.745168 4676 raft_consensus.cc:729] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4b3500f5e74bb996a7cf42208aefaf, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.745496 4676 consensus_queue.cc:260] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.746032 4681 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:23.746464 4681 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:23.747002 4676 ts_tablet_manager.cc:1428] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 11:16:23.754284 4675 tablet_bootstrap.cc:492] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: No bootstrap required, opened a new log
I20260807 11:16:23.754688 4675 ts_tablet_manager.cc:1397] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20260807 11:16:23.755602 4675 raft_consensus.cc:348] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.755911 4675 raft_consensus.cc:374] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:23.756100 4675 raft_consensus.cc:729] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad8ea741754c708550d8baeded1728, State: Initialized, Role: FOLLOWER
I20260807 11:16:23.756489 4675 consensus_queue.cc:260] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.757333 4694 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:23.757745 4694 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:23.758004 4675 ts_tablet_manager.cc:1428] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 11:16:23.957363 4681 raft_consensus.cc:482] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:23.957590 4681 raft_consensus.cc:504] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.958302 4681 leader_election.cc:288] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067), 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821)
I20260807 11:16:23.958947 4302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44a36113a9d49bca600b61f8c9b9ba5" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad8ea741754c708550d8baeded1728" is_pre_election: true
I20260807 11:16:23.959014 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44a36113a9d49bca600b61f8c9b9ba5" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469" is_pre_election: true
I20260807 11:16:23.959257 4302 raft_consensus.cc:2454] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 0.
I20260807 11:16:23.959285 4453 raft_consensus.cc:2454] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 0.
I20260807 11:16:23.959710 4526 leader_election.cc:302] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b3500f5e74bb996a7cf42208aefaf, 782b04af5c054992b6a58a7916e73469; no voters:
I20260807 11:16:23.960059 4681 raft_consensus.cc:2781] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:23.960253 4681 raft_consensus.cc:482] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:23.960438 4681 raft_consensus.cc:3037] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.963500 4681 raft_consensus.cc:504] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.964464 4681 leader_election.cc:288] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 election: Requested vote from peers f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067), 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821)
I20260807 11:16:23.964790 4302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44a36113a9d49bca600b61f8c9b9ba5" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad8ea741754c708550d8baeded1728"
I20260807 11:16:23.965024 4302 raft_consensus.cc:3037] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.965008 4453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44a36113a9d49bca600b61f8c9b9ba5" candidate_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782b04af5c054992b6a58a7916e73469"
I20260807 11:16:23.965281 4453 raft_consensus.cc:3037] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:23.967448 4302 raft_consensus.cc:2454] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 1.
I20260807 11:16:23.967888 4528 leader_election.cc:302] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b3500f5e74bb996a7cf42208aefaf, f5ad8ea741754c708550d8baeded1728; no voters:
I20260807 11:16:23.968297 4681 raft_consensus.cc:2781] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:23.968524 4681 raft_consensus.cc:686] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 LEADER]: Becoming Leader. State: Replica: 4b4b3500f5e74bb996a7cf42208aefaf, State: Running, Role: LEADER
I20260807 11:16:23.968840 4453 raft_consensus.cc:2454] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4b3500f5e74bb996a7cf42208aefaf in term 1.
I20260807 11:16:23.968834 4681 consensus_queue.cc:237] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } }
I20260807 11:16:23.970768 4707 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: TxnStatusTablet state changed. Reason: New leader 4b4b3500f5e74bb996a7cf42208aefaf. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:23.971449 4707 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:23.972548 4721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:16:23.972967 4721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:16:23.976222 4136 catalog_manager.cc:5495] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131). New cstate: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } health_report { overall_health: HEALTHY } } }
I20260807 11:16:24.044272 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.033034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.2.223.131:36003)
0807 11:16:24.034334 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.034350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.034586 (+ 236us) server_negotiation.cc:407] Connection header received
0807 11:16:24.034942 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.034945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.035031 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.035303 (+ 272us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:24.035987 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.037316 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.040077 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.040865 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.040968 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.042335 (+ 1367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:24.043099 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:24.043225 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.043579 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.043711 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":865,"threads_started":1}
I20260807 11:16:24.044239 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.033684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:36003 (local address 127.0.0.1:53716)
0807 11:16:24.034433 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:24.034459 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:24.034692 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:24.035385 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:24.035391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:24.035406 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:24.035769 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:24.035786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.037471 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.037477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:24.039816 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:24.039846 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.041124 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.041148 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:24.041745 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.041896 (+ 151us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:24.043303 (+ 1407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:24.043328 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:16:24.043515 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 11:16:24.043756 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525}
I20260807 11:16:24.052043 4302 raft_consensus.cc:1261] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Refusing update from remote peer 4b4b3500f5e74bb996a7cf42208aefaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:24.052198 4453 raft_consensus.cc:1261] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Refusing update from remote peer 4b4b3500f5e74bb996a7cf42208aefaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:24.053167 4707 consensus_queue.cc:1035] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:16:24.054107 4681 consensus_queue.cc:1035] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:24.060111 4680 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: TxnStatusTablet state changed. Reason: New leader 4b4b3500f5e74bb996a7cf42208aefaf. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.060566 4680 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:24.064730 4680 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.065069 4680 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 782b04af5c054992b6a58a7916e73469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:24.067874 4725 mvcc.cc:205] Tried to move back new op lower bound from 7315871269066698752 to 7315871268739366912. Current Snapshot: MvccSnapshot[applied={T|T < 7315871269066698752}]
I20260807 11:16:24.071729 4694 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: TxnStatusTablet state changed. Reason: New leader 4b4b3500f5e74bb996a7cf42208aefaf. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.072252 4694 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:24.072400 4681 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.072956 4681 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:24.074388 4681 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.074827 4681 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P 4b4b3500f5e74bb996a7cf42208aefaf: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:24.080916 4694 tablet_replica.cc:439] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 } } peers { permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 } } peers { permanent_uuid: "4b4b3500f5e74bb996a7cf42208aefaf" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 36003 } } }
I20260807 11:16:24.081219 4694 tablet_replica.cc:442] T a44a36113a9d49bca600b61f8c9b9ba5 P f5ad8ea741754c708550d8baeded1728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:24.087226 4681 consensus_queue.cc:1035] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ad8ea741754c708550d8baeded1728" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:24.101181 4681 consensus_queue.cc:1035] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "782b04af5c054992b6a58a7916e73469" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 34821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:24.110035 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.088835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.2.223.131:36003)
0807 11:16:24.089100 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.089107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.095264 (+ 6157us) server_negotiation.cc:407] Connection header received
0807 11:16:24.095557 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.095562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.095643 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.095745 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:24.096946 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.098268 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.106768 (+ 8500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.108219 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.108403 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.109092 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:24.109243 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:24.109333 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.109572 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.109670 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:16:24.134588 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.124741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:59719 (local address 127.2.223.131:36003)
0807 11:16:24.124934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.124938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.126543 (+ 1605us) server_negotiation.cc:407] Connection header received
0807 11:16:24.126831 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.126841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.126941 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.127076 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:24.128000 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.129339 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.131897 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.132584 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.132734 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.133289 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:24.133385 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:24.133529 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.133677 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.133749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 11:16:24.134874 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.125224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:36003 (local address 127.2.223.131:59719)
0807 11:16:24.126421 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:24.126437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:24.126663 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:24.127089 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:24.127094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:24.127106 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:24.127781 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:24.127792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.129589 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.129595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:24.131673 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:24.131686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.132791 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.132800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:24.133013 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.133065 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:24.133529 (+ 464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:24.133543 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 11:16:24.133647 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:16:24.133796 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":3,"thread_start_us":651,"threads_started":1}
I20260807 11:16:24.226974 4136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:24.227249 4136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:24.231225 4136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515be0e433544537811c4abf9877f564: Sending DeleteTablet for 3 replicas of tablet f1d801c2916340ea9b680af011dec4c2
I20260807 11:16:24.232146 4281 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d801c2916340ea9b680af011dec4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:24 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:39790
I20260807 11:16:24.232532 4427 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d801c2916340ea9b680af011dec4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:24 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260807 11:16:24.232889 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d801c2916340ea9b680af011dec4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:24 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:53714
I20260807 11:16:24.233721 4745 tablet_replica.cc:330] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: stopping tablet replica
I20260807 11:16:24.234292 4745 raft_consensus.cc:2229] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:24.234661 4745 raft_consensus.cc:2258] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:24.236761 4745 ts_tablet_manager.cc:1905] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:24.244727 4746 tablet_replica.cc:330] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: stopping tablet replica
I20260807 11:16:24.245891 4746 raft_consensus.cc:2229] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:24.246995 4746 raft_consensus.cc:2258] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:24.247145 4747 tablet_replica.cc:330] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: stopping tablet replica
I20260807 11:16:24.247543 4747 raft_consensus.cc:2229] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:24.248620 4747 raft_consensus.cc:2258] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:24.249409 4745 ts_tablet_manager.cc:1918] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:24.249838 4745 log.cc:1196] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/f1d801c2916340ea9b680af011dec4c2
I20260807 11:16:24.250836 4745 ts_tablet_manager.cc:1939] T f1d801c2916340ea9b680af011dec4c2 P f5ad8ea741754c708550d8baeded1728: Deleting consensus metadata
I20260807 11:16:24.252471 4122 catalog_manager.cc:4841] TS f5ad8ea741754c708550d8baeded1728 (127.2.223.129:46067): tablet f1d801c2916340ea9b680af011dec4c2 (table test-table [id=4a6dd22c04de4228947e5a00ff7d46df]) successfully deleted
I20260807 11:16:24.253026 4746 ts_tablet_manager.cc:1905] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:24.259338 4747 ts_tablet_manager.cc:1905] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:24.266304 4746 ts_tablet_manager.cc:1918] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:24.266776 4746 log.cc:1196] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/f1d801c2916340ea9b680af011dec4c2
I20260807 11:16:24.267587 4746 ts_tablet_manager.cc:1939] T f1d801c2916340ea9b680af011dec4c2 P 782b04af5c054992b6a58a7916e73469: Deleting consensus metadata
I20260807 11:16:24.267628 4747 ts_tablet_manager.cc:1918] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:24.267931 4747 log.cc:1196] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/f1d801c2916340ea9b680af011dec4c2
I20260807 11:16:24.268697 4747 ts_tablet_manager.cc:1939] T f1d801c2916340ea9b680af011dec4c2 P 4b4b3500f5e74bb996a7cf42208aefaf: Deleting consensus metadata
I20260807 11:16:24.269013 4120 catalog_manager.cc:4841] TS 782b04af5c054992b6a58a7916e73469 (127.2.223.130:34821): tablet f1d801c2916340ea9b680af011dec4c2 (table test-table [id=4a6dd22c04de4228947e5a00ff7d46df]) successfully deleted
I20260807 11:16:24.269950 4122 catalog_manager.cc:4841] TS 4b4b3500f5e74bb996a7cf42208aefaf (127.2.223.131:36003): tablet f1d801c2916340ea9b680af011dec4c2 (table test-table [id=4a6dd22c04de4228947e5a00ff7d46df]) successfully deleted
Aug 07 11:16:24 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:24.286276 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.248275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.2.223.129:46067)
0807 11:16:24.263473 (+ 15198us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.263486 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.263505 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:16:24.263565 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.263569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.263663 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.263809 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:24.266996 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.268339 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.270657 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.271163 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.271315 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.277390 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:24.277407 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:24.277422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:24.277455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:24.282076 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.283132 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.283144 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.283148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.283456 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.283987 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.283992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.283995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.284493 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:24.284927 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.285279 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.285768 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14867,"thread_start_us":14742,"threads_started":1}
W20260807 11:16:24.287407 4350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814
I20260807 11:16:24.317443 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.290748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.2.223.190:41321)
0807 11:16:24.295277 (+ 4529us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.295282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.295297 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:16:24.295346 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.295350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.295407 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.295669 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:24.298610 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.299888 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.302690 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.303435 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.303542 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.308296 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:24.308301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:24.308304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:24.308338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:24.313597 (+ 5259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.314510 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.314516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.314519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.314776 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.315195 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.315201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.315215 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.315673 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:24.316121 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.316359 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.316705 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4373,"thread_start_us":4111,"threads_started":1}
W20260807 11:16:24.318320 4136 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46348
I20260807 11:16:24.320736 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4212
I20260807 11:16:24.342955 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.323086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.2.223.129:46067)
0807 11:16:24.323329 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.323336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.328892 (+ 5556us) server_negotiation.cc:407] Connection header received
0807 11:16:24.329116 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.329120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.329183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.329290 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:24.330620 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.331299 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.332475 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.333025 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.333180 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.335905 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:24.335910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:24.335913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:24.335944 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:24.339760 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.340519 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.340524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.340526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.340676 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.341135 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.341138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.341140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.341413 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:24.341690 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.342136 (+ 446us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.342462 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:16:24.343637 4350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39816
W20260807 11:16:24.344365 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:24.344631 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4212
I20260807 11:16:24.382998 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4360
Aug 07 11:16:24 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:24.403723 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.383945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.2.223.130:34821)
0807 11:16:24.384676 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.384681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.384697 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:16:24.384753 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.384756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.384846 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.384983 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:24.386752 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.387936 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.389702 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.390396 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.390570 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.395968 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:24.395982 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:24.395998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:24.396038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:24.400179 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.401003 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.401012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.401014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.401201 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.401630 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.401634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.401638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.402038 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:24.402328 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.402947 (+ 619us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.403209 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
W20260807 11:16:24.404496 4497 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46710
W20260807 11:16:24.405220 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4360: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:24.405405 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4360
I20260807 11:16:24.428181 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4509
Aug 07 11:16:24 dist-test-slave-5l3k krb5kdc[4091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:24.444269 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:24.428985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.2.223.131:36003)
0807 11:16:24.429149 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:24.429154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:24.429385 (+ 231us) server_negotiation.cc:407] Connection header received
0807 11:16:24.429612 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:24.429619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:24.429696 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:24.429822 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:24.431132 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.431893 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.433111 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:24.433586 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:24.433733 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:24.437966 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:24.437977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:24.437991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:24.438022 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:24.441161 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.441841 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.441856 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.441858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.442025 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:24.442404 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:24.442410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:24.442413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:24.442939 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:24.443309 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:24.443630 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 11:16:24.443874 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260807 11:16:24.444957 4646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53732
W20260807 11:16:24.445685 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4509: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:24.445840 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4509
I20260807 11:16:24.469223 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4104
I20260807 11:16:24.470445 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:24.591033 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4104
2026-08-07T11:16:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5210 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:26 dist-test-slave-5l3k krb5kdc[4759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:26 dist-test-slave-5l3k krb5kdc[4759](info): set up 2 sockets
Aug 07 11:16:26 dist-test-slave-5l3k krb5kdc[4759](info): commencing operation
krb5kdc: starting...
W20260807 11:16:27.287636 2942 mini_kdc.cc:121] Time spent starting KDC: real 2.648s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:27 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:27.495877 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35657
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:27.823987 4774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:27.824275 4774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:27.824429 4774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:27.831633 4774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:27.831732 4774 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:16:27.831799 4774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:27.831890 4774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:27.831969 4774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:27.832057 4774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:27.841854 4774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:27.843240 4774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:27.844630 4774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:27.856980 4779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:27.857084 4780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:27.857074 4782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:27.858230 4774 server_base.cc:1034] running on GCE node
I20260807 11:16:27.858990 4774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:27.860343 4774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:27.861562 4774 hybrid_clock.cc:648] HybridClock initialized: now 1786101387861497 us; error 36 us; skew 500 ppm
I20260807 11:16:27.861840 4774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:27 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101387, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:27.868727 4774 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:27.870256 4774 webserver.cc:460] Webserver started at http://127.2.223.190:40309/ using document root <none> and password file <none>
I20260807 11:16:27.871006 4774 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:27.871100 4774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:27.871358 4774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:27.874290 4774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "79cc3da98fc74d36b42f79836ab86378"
format_stamp: "Formatted at 2026-08-07 11:16:27 on dist-test-slave-5l3k"
server_key: "d26d5f9f6fcc2a7715d090926e3ef9dd"
server_key_iv: "f97fa73b9e9aaa708b7b38a8ae8f5da8"
server_key_version: "encryptionkey@0"
I20260807 11:16:27.875202 4774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "79cc3da98fc74d36b42f79836ab86378"
format_stamp: "Formatted at 2026-08-07 11:16:27 on dist-test-slave-5l3k"
server_key: "d26d5f9f6fcc2a7715d090926e3ef9dd"
server_key_iv: "f97fa73b9e9aaa708b7b38a8ae8f5da8"
server_key_version: "encryptionkey@0"
I20260807 11:16:27.880453 4774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:27.883811 4789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:27.885093 4774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:27.885274 4774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "79cc3da98fc74d36b42f79836ab86378"
format_stamp: "Formatted at 2026-08-07 11:16:27 on dist-test-slave-5l3k"
server_key: "d26d5f9f6fcc2a7715d090926e3ef9dd"
server_key_iv: "f97fa73b9e9aaa708b7b38a8ae8f5da8"
server_key_version: "encryptionkey@0"
I20260807 11:16:27.885478 4774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:27.906040 4774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:27.910213 4774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:27.910578 4774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:27.932598 4774 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:35657
I20260807 11:16:27.932636 4857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:35657 every 8 connection(s)
I20260807 11:16:27.933959 4774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:27.936337 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4774
I20260807 11:16:27.936565 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:27.937422 2942 external_mini_cluster.cc:1428] Setting key f84775b545e6005d3ffabab84414d3f7
I20260807 11:16:27.939266 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:16:27 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:27.953642 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Bootstrap starting.
I20260807 11:16:27.956790 4858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:27.958120 4858 log.cc:824] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:27.961146 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:27.939904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.2.223.190:35657)
0807 11:16:27.940717 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:27.940768 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:27.940822 (+ 54us) server_negotiation.cc:407] Connection header received
0807 11:16:27.942403 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:27.942425 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:27.942933 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:27.943268 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:27.945147 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:27.946559 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:27.948289 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:27.948925 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:27.949050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:27.953674 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:27.953682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:27.953695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:27.953734 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:27.956851 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:27.957654 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:27.957660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:27.957665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:27.957868 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:27.958351 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:27.958354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:27.958356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:27.958968 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:27.959294 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:27.959745 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 11:16:27.960161 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":408,"threads_started":1}
I20260807 11:16:27.961190 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: No bootstrap required, opened a new log
I20260807 11:16:27.969558 4858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } }
I20260807 11:16:27.969980 4858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:27.970093 4858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc3da98fc74d36b42f79836ab86378, State: Initialized, Role: FOLLOWER
I20260807 11:16:27.970682 4858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } }
I20260807 11:16:27.970968 4858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:27.971177 4858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:27.971374 4858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:27.973567 4858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } }
I20260807 11:16:27.974064 4858 leader_election.cc:302] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc3da98fc74d36b42f79836ab86378; no voters:
I20260807 11:16:27.974735 4858 leader_election.cc:288] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:27.974864 4863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:27.975569 4863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc3da98fc74d36b42f79836ab86378, State: Running, Role: LEADER
I20260807 11:16:27.976182 4863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } }
I20260807 11:16:27.976820 4858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:27.979192 4864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79cc3da98fc74d36b42f79836ab86378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } } }
I20260807 11:16:27.979579 4864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:27.980083 4865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cc3da98fc74d36b42f79836ab86378. Latest consensus state: current_term: 1 leader_uuid: "79cc3da98fc74d36b42f79836ab86378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc3da98fc74d36b42f79836ab86378" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35657 } } }
I20260807 11:16:27.980993 4865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:27.983517 4869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:27.989777 4869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:27.998230 4869 catalog_manager.cc:1347] Generated new cluster ID: 23fe678dabfc498cac63c0f68c0df62f
I20260807 11:16:27.998345 4869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:28.010607 4869 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:28.012502 4869 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:28.023926 4869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Generated new TSK 0
I20260807 11:16:28.024930 4869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:28.078999 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35657
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:28.412856 4886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:28.413183 4886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:28.413367 4886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:28.420603 4886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:28.420754 4886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:28.420903 4886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:16:28.421001 4886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:28.421180 4886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:28.430691 4886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:35657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:28.431921 4886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:28.433210 4886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:28.447211 4891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:28.447263 4894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:28.447234 4892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:28.448136 4886 server_base.cc:1034] running on GCE node
I20260807 11:16:28.448777 4886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:28.449743 4886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:28.450958 4886 hybrid_clock.cc:648] HybridClock initialized: now 1786101388450885 us; error 64 us; skew 500 ppm
I20260807 11:16:28.451292 4886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:28 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:28.454591 4886 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:28.456034 4886 webserver.cc:460] Webserver started at http://127.2.223.129:36529/ using document root <none> and password file <none>
I20260807 11:16:28.456651 4886 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:28.456753 4886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:28.457000 4886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:28.459931 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "5137a6b0550f436894a08df47d21c72d"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "293fdb59f566d08f18d92f6294e41dd0"
server_key_iv: "9caa2174ba9f41fe9dcc878e6655fc2a"
server_key_version: "encryptionkey@0"
I20260807 11:16:28.460741 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "5137a6b0550f436894a08df47d21c72d"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "293fdb59f566d08f18d92f6294e41dd0"
server_key_iv: "9caa2174ba9f41fe9dcc878e6655fc2a"
server_key_version: "encryptionkey@0"
I20260807 11:16:28.465803 4886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 11:16:28.469286 4901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:28.470692 4886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:28.470878 4886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "5137a6b0550f436894a08df47d21c72d"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "293fdb59f566d08f18d92f6294e41dd0"
server_key_iv: "9caa2174ba9f41fe9dcc878e6655fc2a"
server_key_version: "encryptionkey@0"
I20260807 11:16:28.471107 4886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:28.488018 4886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:28.492560 4886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:28.492900 4886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:28.495621 4886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:28.495761 4886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:28.495905 4886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:28.496028 4886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:28 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:28.522001 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:28.495894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:55441 (local address 127.2.223.190:35657)
0807 11:16:28.496472 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:28.496478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:28.497861 (+ 1383us) server_negotiation.cc:407] Connection header received
0807 11:16:28.501033 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:28.501042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:28.501145 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:28.501257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:28.503707 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.505041 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.507578 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.508251 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.508394 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:28.514121 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:28.514129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:28.514136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:28.514190 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:28.517668 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:28.518520 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:28.518528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:28.518536 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:28.518664 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:28.519296 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:28.519308 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:28.519311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:28.519746 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:28.519905 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:28.521052 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0807 11:16:28.521415 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":298,"threads_started":1}
I20260807 11:16:28.522684 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:28.496687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.129:55441)
0807 11:16:28.497645 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:28.497701 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:28.500638 (+ 2937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:28.501653 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:28.501667 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:28.502545 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:28.503459 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:28.503497 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.505286 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.505292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:28.507418 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:28.507432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.508623 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.508637 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:28.508959 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:28.509728 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:28.509752 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:28.513746 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:28.517851 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:28.517861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:28.517883 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:28.518349 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:28.518789 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:28.518798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:28.518801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:28.519159 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:28.519979 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:28.519987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:28.520578 (+ 591us) client_negotiation.cc:769] Sending connection context
0807 11:16:28.520845 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 11:16:28.521296 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":358,"threads_started":1}
I20260807 11:16:28.545117 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:28.528933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:56421 (local address 127.2.223.190:35657)
0807 11:16:28.529119 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:28.529127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:28.529974 (+ 847us) server_negotiation.cc:407] Connection header received
0807 11:16:28.530179 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:28.530185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:28.530269 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:28.530377 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:28.531691 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.532929 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.534568 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.535332 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.535492 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:28.538032 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:28.538042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:28.538055 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:28.538105 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:28.541965 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:28.542786 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:28.542795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:28.542799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:28.542984 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:28.543521 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:28.543525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:28.543527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:28.543934 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:28.544075 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:28.544686 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 11:16:28.544868 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 11:16:28.545714 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:28.529022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.129:56421)
0807 11:16:28.529870 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:28.529887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:28.530049 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:28.530404 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:28.530409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:28.530992 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:28.531419 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:28.531438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.533125 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.533131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:28.534388 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:28.534402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:28.535570 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:28.535582 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:28.535777 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:28.536478 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:28.536507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:28.537818 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:28.542145 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:28.542149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:28.542153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:28.542655 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:28.543136 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:28.543144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:28.543150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:28.543371 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:28.544065 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:28.544073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:28.544364 (+ 291us) client_negotiation.cc:769] Sending connection context
0807 11:16:28.544559 (+ 195us) client_negotiation.cc:240] Negotiation successful
0807 11:16:28.545024 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":405,"threads_started":1}
I20260807 11:16:28.552754 4886 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:35489
I20260807 11:16:28.552795 5028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:35489 every 8 connection(s)
I20260807 11:16:28.554021 4886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:28.554401 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 4886
I20260807 11:16:28.554636 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:28.555562 2942 external_mini_cluster.cc:1428] Setting key 0315f173df4cfaa532f30548bece37fa
I20260807 11:16:28.556396 5030 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35657
I20260807 11:16:28.556643 5030 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:28.557457 5030 heartbeater.cc:511] Master 127.2.223.190:35657 requested a full tablet report, sending...
I20260807 11:16:28.559912 4808 ts_manager.cc:194] Registered new tserver with Master: 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489)
I20260807 11:16:28.562093 4808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:55441
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:28.609212 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35657
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:28.942916 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:28.943223 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:28.943418 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:28.950452 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:28.950632 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:28.950785 5034 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:16:28.950887 5034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:28.951020 5034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:28.961095 5034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:35657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:28.962769 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:28.964140 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:28.978572 5043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:28.978652 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:28.978626 5041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:28.979446 5034 server_base.cc:1034] running on GCE node
I20260807 11:16:28.980085 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:28.981019 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:28.982262 5034 hybrid_clock.cc:648] HybridClock initialized: now 1786101388982210 us; error 49 us; skew 500 ppm
I20260807 11:16:28.982642 5034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:28 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:28.985949 5034 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:28.987474 5034 webserver.cc:460] Webserver started at http://127.2.223.130:34353/ using document root <none> and password file <none>
I20260807 11:16:28.988155 5034 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:28.988301 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:28.988674 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:28.991780 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "a303789002d94d4794f96859c10f8e44"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "deced17ef89dc39a106f81ae074ec20f"
server_key_iv: "34ce675bf445a29094d5e3e11d11f2e5"
server_key_version: "encryptionkey@0"
I20260807 11:16:28.992553 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "a303789002d94d4794f96859c10f8e44"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "deced17ef89dc39a106f81ae074ec20f"
server_key_iv: "34ce675bf445a29094d5e3e11d11f2e5"
server_key_version: "encryptionkey@0"
I20260807 11:16:28.997740 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:16:29.001250 5050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:29.002674 5034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:29.002846 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "a303789002d94d4794f96859c10f8e44"
format_stamp: "Formatted at 2026-08-07 11:16:28 on dist-test-slave-5l3k"
server_key: "deced17ef89dc39a106f81ae074ec20f"
server_key_iv: "34ce675bf445a29094d5e3e11d11f2e5"
server_key_version: "encryptionkey@0"
I20260807 11:16:29.003036 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:29.022418 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:29.027000 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:29.027294 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:29.029996 5034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:29.030138 5034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:29.030267 5034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:29.030385 5034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101388, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:29.058146 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.031208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.130:57317)
0807 11:16:29.032103 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.032171 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.034976 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.035921 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.035935 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.036969 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:29.038068 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.038100 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.039356 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.039364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.042286 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.042308 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.043666 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.043679 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.043955 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.044885 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:29.044926 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:29.048907 (+ 3981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:29.052557 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.052567 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.052592 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.053395 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.053972 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.053983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.053989 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.054275 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.055036 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:29.055049 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:29.055573 (+ 524us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.055912 (+ 339us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.056497 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":369,"threads_started":1}
I20260807 11:16:29.059316 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.030506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:57317 (local address 127.2.223.190:35657)
0807 11:16:29.030756 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.030765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.032301 (+ 1536us) server_negotiation.cc:407] Connection header received
0807 11:16:29.035329 (+ 3028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.035336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.035458 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.035603 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.038279 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.039160 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.042507 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.043342 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.043497 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.049257 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.049266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.049272 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.049321 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.052391 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.053604 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.053614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.053620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.053791 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.054404 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.054412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.054415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.054862 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.055038 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.058742 (+ 3704us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.059002 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:16:29.078330 5034 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:40939
I20260807 11:16:29.078388 5175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:40939 every 8 connection(s)
I20260807 11:16:29.079977 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:29.081629 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5034
I20260807 11:16:29.081813 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:29.082281 2942 external_mini_cluster.cc:1428] Setting key f4e4fb54d2b7e9b03a45ab842d64e825
I20260807 11:16:29.082808 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.065173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48721 (local address 127.2.223.190:35657)
0807 11:16:29.065406 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.065415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.066033 (+ 618us) server_negotiation.cc:407] Connection header received
0807 11:16:29.066363 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.066368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.066435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.066617 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.067636 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.068446 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.070551 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.071372 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.071518 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.074593 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.074603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.074611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.074662 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.078883 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.079684 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.079690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.079693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.079863 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.080387 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.080395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.080401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.080908 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.082194 (+ 1286us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.082296 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.082561 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:16:29.082093 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.065167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.130:48721)
0807 11:16:29.065918 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.065944 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.066195 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.066647 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.066652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.067181 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:29.067486 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.067500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.068646 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.068650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.070341 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.070366 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.071623 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.071632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.071853 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.072738 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:29.072777 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:29.074336 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:29.079080 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.079085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.079088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.079554 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.080081 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.080088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.080094 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.080262 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.081097 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:29.081105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:29.081318 (+ 213us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.081444 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.081664 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":383,"threads_started":1}
I20260807 11:16:29.083871 5176 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35657
I20260807 11:16:29.084100 5176 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:29.085049 5176 heartbeater.cc:511] Master 127.2.223.190:35657 requested a full tablet report, sending...
I20260807 11:16:29.086445 4807 ts_manager.cc:194] Registered new tserver with Master: a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939)
I20260807 11:16:29.087739 4807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:57317
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:29.129954 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35657
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:29.459721 5180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:29.459987 5180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:29.460106 5180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:29.467263 5180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:29.467370 5180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:29.467463 5180 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:16:29.467535 5180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:16:29.467602 5180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:29.477192 5180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:35657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:29.478585 5180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:29.479888 5180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:29.492836 5185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:29.493099 5186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:29.493134 5188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:29.494105 5180 server_base.cc:1034] running on GCE node
I20260807 11:16:29.494848 5180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:29.495806 5180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:29.497006 5180 hybrid_clock.cc:648] HybridClock initialized: now 1786101389496963 us; error 38 us; skew 500 ppm
I20260807 11:16:29.497347 5180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:29.500650 5180 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:29.502106 5180 webserver.cc:460] Webserver started at http://127.2.223.131:45333/ using document root <none> and password file <none>
I20260807 11:16:29.502861 5180 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:29.503022 5180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:29.503353 5180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:29.506340 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "2cb719d93353476bb464f6a78181b41b"
format_stamp: "Formatted at 2026-08-07 11:16:29 on dist-test-slave-5l3k"
server_key: "cf4972cafcaeb71a9b22e24c1ec9b3a7"
server_key_iv: "0a2b30a8bda5eb70bc0507d76c8dc804"
server_key_version: "encryptionkey@0"
I20260807 11:16:29.507198 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb719d93353476bb464f6a78181b41b"
format_stamp: "Formatted at 2026-08-07 11:16:29 on dist-test-slave-5l3k"
server_key: "cf4972cafcaeb71a9b22e24c1ec9b3a7"
server_key_iv: "0a2b30a8bda5eb70bc0507d76c8dc804"
server_key_version: "encryptionkey@0"
I20260807 11:16:29.512398 5180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:29.515796 5195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:29.517107 5180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:29.517284 5180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "2cb719d93353476bb464f6a78181b41b"
format_stamp: "Formatted at 2026-08-07 11:16:29 on dist-test-slave-5l3k"
server_key: "cf4972cafcaeb71a9b22e24c1ec9b3a7"
server_key_iv: "0a2b30a8bda5eb70bc0507d76c8dc804"
server_key_version: "encryptionkey@0"
I20260807 11:16:29.517479 5180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:29.534924 5180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:29.539240 5180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:29.539541 5180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:29.542138 5180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:29.542279 5180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:29.542426 5180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:29.542531 5180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:29.565662 5030 heartbeater.cc:503] Master 127.2.223.190:35657 was elected leader, sending a full tablet report...
I20260807 11:16:29.565834 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.543450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.131:38447)
0807 11:16:29.544339 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.544395 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.547357 (+ 2962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.548598 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.548617 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.549733 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:29.550885 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.550919 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.552373 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.552379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.553682 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.553694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.554527 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.554536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.554741 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.555405 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:29.555425 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:29.557610 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:29.561220 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.561228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.561249 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.561703 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.562129 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.562134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.562138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.562319 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.563018 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:29.563026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:29.563418 (+ 392us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.563635 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.564085 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":345,"threads_started":1}
I20260807 11:16:29.567143 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.543007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:38447 (local address 127.2.223.190:35657)
0807 11:16:29.543198 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.543207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.544541 (+ 1334us) server_negotiation.cc:407] Connection header received
0807 11:16:29.547763 (+ 3222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.547771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.547870 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.547972 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.551140 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.552182 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.553843 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.554270 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.554407 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.557953 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.557958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.557961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.557999 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.561081 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.561858 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.561865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.561874 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.562006 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.562445 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.562457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.562459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.562822 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.564182 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.564423 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.566670 (+ 2247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:16:29.593669 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.574534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:40979 (local address 127.2.223.190:35657)
0807 11:16:29.574747 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.574755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.579266 (+ 4511us) server_negotiation.cc:407] Connection header received
0807 11:16:29.579469 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.579473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.579534 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.579664 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.581163 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.582255 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.584245 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.584835 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.584946 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.587281 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.587289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.587297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.587353 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.591011 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.591662 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.591669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.591672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.591824 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.592203 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.592208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.592211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.592486 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.592651 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.593076 (+ 425us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.593310 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:16:29.593768 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.574556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.2.223.131:40979)
0807 11:16:29.579127 (+ 4571us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.579153 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.579344 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.579699 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.579703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.580403 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:29.580933 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.580963 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.582425 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.582430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.584058 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.584072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.585279 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.585284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.585449 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.586235 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:29.586253 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:29.587075 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:29.591154 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.591159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.591163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.591539 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.591940 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.591945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.591948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.592080 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.592630 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:29.592634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:29.592816 (+ 182us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.592964 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.593290 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4371,"thread_start_us":4097,"threads_started":1}
I20260807 11:16:29.607131 5180 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:41335
I20260807 11:16:29.607180 5323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:41335 every 8 connection(s)
I20260807 11:16:29.608317 5180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:29.609266 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5180
I20260807 11:16:29.609463 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:29.610075 2942 external_mini_cluster.cc:1428] Setting key e56358e0d6849d30b108c86634e3998d
I20260807 11:16:29.610723 5324 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35657
I20260807 11:16:29.610987 5324 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:29.611742 5324 heartbeater.cc:511] Master 127.2.223.190:35657 requested a full tablet report, sending...
I20260807 11:16:29.613162 4807 ts_manager.cc:194] Registered new tserver with Master: 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:29.613971 4807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:38447
I20260807 11:16:29.625620 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:29.684930 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.664014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.2.223.190:35657)
0807 11:16:29.664266 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.664274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.664395 (+ 121us) server_negotiation.cc:407] Connection header received
0807 11:16:29.664545 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.664550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.664627 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.664714 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.666053 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.666870 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.668025 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.668456 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.668548 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.680052 (+ 11504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.680058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.680061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.680113 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.682602 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.683173 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.683177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.683179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.683296 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.683600 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.683603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.683605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.683873 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.684062 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.684370 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.684548 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 11:16:29.688998 4807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:29.696116 4807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:29.721853 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.708128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.2.223.130:40939)
0807 11:16:29.709704 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.709710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.709732 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:16:29.709883 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.709895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.710150 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.710367 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:29.711608 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.713519 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.717191 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.719143 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.719316 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.719597 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.721166 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.721404 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":1158,"threads_started":1}
I20260807 11:16:29.722465 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.707920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40939 (local address 127.0.0.1:37002)
0807 11:16:29.708642 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.708682 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.708854 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.710745 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.710756 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.710798 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:29.711347 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.711363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.713743 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.713752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.716945 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.716967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.719908 (+ 2941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.719921 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.720250 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.720323 (+ 73us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.720495 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.720760 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":339,"threads_started":1}
I20260807 11:16:29.728241 5102 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 (DEFAULT_TABLE table=test-table [id=83da28ec70a7415496d9f0fe08f89c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:29.729735 5102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3365ec87f848949a2e15a1806f47f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.731155 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.710759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.2.223.131:41335)
0807 11:16:29.723251 (+ 12492us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.723263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.724095 (+ 832us) server_negotiation.cc:407] Connection header received
0807 11:16:29.724203 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.724210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.724427 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.724681 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:29.725389 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.726877 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.728746 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.730020 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.730185 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.730410 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.730685 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.730827 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12280,"thread_start_us":12091,"threads_started":1}
I20260807 11:16:29.731397 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.710036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41335 (local address 127.0.0.1:34820)
0807 11:16:29.723743 (+ 13707us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.723768 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.723933 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.724737 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.724741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.724782 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:29.725129 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.725140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.727004 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.727011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.728549 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.728565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.730236 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.730242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.730412 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.730450 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.730634 (+ 184us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.730770 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13561,"mutex_wait_us":51,"thread_start_us":13832,"threads_started":1}
I20260807 11:16:29.731503 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.708074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:35489 (local address 127.0.0.1:50292)
0807 11:16:29.722808 (+ 14734us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.722833 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.722970 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.724196 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.724202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.724229 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:29.724384 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.724395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.726298 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.726304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.728125 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.728140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.730475 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.730481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.730741 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.730775 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.730865 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.731021 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14601,"mutex_wait_us":1,"thread_start_us":15397,"threads_started":1}
I20260807 11:16:29.732173 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.708703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.2.223.129:35489)
0807 11:16:29.721500 (+ 12797us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.721512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.723124 (+ 1612us) server_negotiation.cc:407] Connection header received
0807 11:16:29.723261 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.723273 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.723601 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.723844 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:29.724600 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.726141 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.728583 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.730042 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.730186 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.730377 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.731730 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.731890 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12574,"thread_start_us":12797,"threads_started":1}
I20260807 11:16:29.734961 5255 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 (DEFAULT_TABLE table=test-table [id=83da28ec70a7415496d9f0fe08f89c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:29.735530 4956 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 (DEFAULT_TABLE table=test-table [id=83da28ec70a7415496d9f0fe08f89c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:29.736421 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3365ec87f848949a2e15a1806f47f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.737172 4956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3365ec87f848949a2e15a1806f47f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.745559 5344 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Bootstrap starting.
I20260807 11:16:29.749882 5344 tablet_bootstrap.cc:654] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.752214 5344 log.cc:824] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:29.755501 5344 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: No bootstrap required, opened a new log
I20260807 11:16:29.755877 5344 ts_tablet_manager.cc:1397] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 11:16:29.757756 5346 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Bootstrap starting.
I20260807 11:16:29.759078 5347 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Bootstrap starting.
I20260807 11:16:29.765553 5346 tablet_bootstrap.cc:654] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.765965 5347 tablet_bootstrap.cc:654] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.766877 5346 log.cc:824] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:29.767637 5347 log.cc:824] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:29.769961 5344 raft_consensus.cc:348] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.770430 5344 raft_consensus.cc:374] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.770571 5344 raft_consensus.cc:729] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303789002d94d4794f96859c10f8e44, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.770679 5346 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: No bootstrap required, opened a new log
I20260807 11:16:29.770967 5346 ts_tablet_manager.cc:1397] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260807 11:16:29.771019 5347 tablet_bootstrap.cc:492] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: No bootstrap required, opened a new log
I20260807 11:16:29.771482 5347 ts_tablet_manager.cc:1397] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260807 11:16:29.772755 5344 consensus_queue.cc:260] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.781141 5176 heartbeater.cc:503] Master 127.2.223.190:35657 was elected leader, sending a full tablet report...
I20260807 11:16:29.782274 5344 ts_tablet_manager.cc:1428] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Time spent starting tablet: real 0.026s user 0.017s sys 0.006s
I20260807 11:16:29.784332 5347 raft_consensus.cc:348] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.784878 5347 raft_consensus.cc:374] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.785006 5347 raft_consensus.cc:729] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.785749 5347 consensus_queue.cc:260] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.787246 5324 heartbeater.cc:503] Master 127.2.223.190:35657 was elected leader, sending a full tablet report...
I20260807 11:16:29.789254 5347 ts_tablet_manager.cc:1428] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Time spent starting tablet: real 0.018s user 0.009s sys 0.006s
I20260807 11:16:29.788437 5346 raft_consensus.cc:348] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.789669 5346 raft_consensus.cc:374] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.789790 5346 raft_consensus.cc:729] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137a6b0550f436894a08df47d21c72d, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.790449 5346 consensus_queue.cc:260] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.792616 5346 ts_tablet_manager.cc:1428] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Time spent starting tablet: real 0.022s user 0.011s sys 0.004s
W20260807 11:16:29.808255 5031 tablet.cc:2369] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:16:29.832144 5177 tablet.cc:2369] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:16:29.860251 5325 tablet.cc:2369] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:29.867820 5352 raft_consensus.cc:482] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:29.868111 5352 raft_consensus.cc:504] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.869858 5352 leader_election.cc:288] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939), 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489)
I20260807 11:16:29.876936 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.870374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40939 (local address 127.2.223.131:43943)
0807 11:16:29.870765 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.870793 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.870966 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.871383 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.871387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.871433 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:29.871630 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.871646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.872887 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.872892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.874881 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.874893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.876224 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.876231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.876426 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.876459 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.876554 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.876696 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":60}
I20260807 11:16:29.877031 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.870374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:43943 (local address 127.2.223.130:40939)
0807 11:16:29.870610 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.870617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.870895 (+ 278us) server_negotiation.cc:407] Connection header received
0807 11:16:29.871152 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.871157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.871248 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.871348 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:29.871772 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.872749 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.875028 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.876073 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.876204 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.876256 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.876612 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.876766 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:16:29.877547 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.870284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:58673 (local address 127.2.223.129:35489)
0807 11:16:29.870529 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.870536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.871303 (+ 767us) server_negotiation.cc:407] Connection header received
0807 11:16:29.871489 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.871495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.871591 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.871688 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:29.872266 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.873163 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.875420 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.876442 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.876615 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.876683 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.877158 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.877302 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:16:29.878077 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.870415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:35489 (local address 127.2.223.131:58673)
0807 11:16:29.871190 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.871206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.871340 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.871732 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.871736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.871762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:29.872112 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.872125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.873314 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.873322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.875241 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.875257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.876726 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.876735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.876914 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.876950 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.877739 (+ 789us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.877851 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":394,"threads_started":1}
I20260807 11:16:29.878172 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3365ec87f848949a2e15a1806f47f0" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303789002d94d4794f96859c10f8e44" is_pre_election: true
I20260807 11:16:29.878813 5125 raft_consensus.cc:2454] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 0.
I20260807 11:16:29.879127 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3365ec87f848949a2e15a1806f47f0" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d" is_pre_election: true
I20260807 11:16:29.879618 5198 leader_election.cc:302] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:29.879726 4984 raft_consensus.cc:2454] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 0.
I20260807 11:16:29.880045 5352 raft_consensus.cc:2781] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:29.880209 5352 raft_consensus.cc:482] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:29.880352 5352 raft_consensus.cc:3037] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:29.882571 5352 raft_consensus.cc:504] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.883132 5352 leader_election.cc:288] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 election: Requested vote from peers a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939), 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489)
I20260807 11:16:29.883606 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3365ec87f848949a2e15a1806f47f0" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303789002d94d4794f96859c10f8e44"
I20260807 11:16:29.883618 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3365ec87f848949a2e15a1806f47f0" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d"
I20260807 11:16:29.883806 5125 raft_consensus.cc:3037] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:29.883826 4984 raft_consensus.cc:3037] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:29.887094 5125 raft_consensus.cc:2454] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 1.
I20260807 11:16:29.887084 4984 raft_consensus.cc:2454] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 1.
I20260807 11:16:29.887482 5198 leader_election.cc:302] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:29.887894 5352 raft_consensus.cc:2781] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:29.888376 5352 raft_consensus.cc:686] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 1 LEADER]: Becoming Leader. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Running, Role: LEADER
I20260807 11:16:29.888814 5352 consensus_queue.cc:237] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.893600 4807 catalog_manager.cc:5495] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb719d93353476bb464f6a78181b41b (127.2.223.131). New cstate: current_term: 1 leader_uuid: "2cb719d93353476bb464f6a78181b41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } health_report { overall_health: HEALTHY } } }
Aug 07 11:16:29 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101387, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:29.951120 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.936909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.2.223.190:35657)
0807 11:16:29.937072 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:29.937076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:29.937701 (+ 625us) server_negotiation.cc:407] Connection header received
0807 11:16:29.937909 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:29.937920 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:29.937998 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:29.938124 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:29.939448 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.940250 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.941553 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.942055 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.942200 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.945643 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:29.945649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:29.945652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:29.945680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:29.948290 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.948994 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.949002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.949006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.949136 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:29.949514 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:29.949521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:29.949524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:29.949830 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:29.949983 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:29.950393 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 11:16:29.950754 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":50}
I20260807 11:16:29.951157 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:29.937014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35657 (local address 127.0.0.1:53148)
0807 11:16:29.937591 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:29.937611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:29.937790 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:29.938141 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:29.938145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:29.938988 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:29.939309 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.939320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.940388 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.940393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.941409 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:29.941421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:29.942247 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:29.942252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:29.942419 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:29.943160 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:29.943177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:29.945458 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:29.948427 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.948440 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.948448 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.948876 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.949265 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:29.949270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:29.949273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:29.949408 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:29.949961 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:29.949969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:29.950142 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 11:16:29.950315 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 11:16:29.950630 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":316,"threads_started":1}
I20260807 11:16:29.956399 4807 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:53148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:16:29.957330 4807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:29.964772 5102 tablet_service.cc:1467] Processing CreateTablet for tablet 165fcf74f49a4fcda5b7eb324c273b0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b79ade7368f4cb19744e34bcd1d5980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:29.964715 4956 tablet_service.cc:1467] Processing CreateTablet for tablet 165fcf74f49a4fcda5b7eb324c273b0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b79ade7368f4cb19744e34bcd1d5980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:29.965024 5255 tablet_service.cc:1467] Processing CreateTablet for tablet 165fcf74f49a4fcda5b7eb324c273b0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b79ade7368f4cb19744e34bcd1d5980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:16:29.965756 5102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fcf74f49a4fcda5b7eb324c273b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.965756 4956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fcf74f49a4fcda5b7eb324c273b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.965806 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fcf74f49a4fcda5b7eb324c273b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:29.972136 5347 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: Bootstrap starting.
I20260807 11:16:29.973192 5346 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: Bootstrap starting.
I20260807 11:16:29.973798 5347 tablet_bootstrap.cc:654] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.975288 5346 tablet_bootstrap.cc:654] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.975584 5344 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: Bootstrap starting.
I20260807 11:16:29.976573 5347 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: No bootstrap required, opened a new log
I20260807 11:16:29.976742 5347 ts_tablet_manager.cc:1397] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260807 11:16:29.977425 5347 raft_consensus.cc:348] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.977739 5347 raft_consensus.cc:374] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.977841 5347 raft_consensus.cc:729] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.977931 5344 tablet_bootstrap.cc:654] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:29.978073 5347 consensus_queue.cc:260] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.978693 5347 ts_tablet_manager.cc:1428] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 11:16:29.979460 5352 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:29.979872 5352 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:29.982398 5344 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: No bootstrap required, opened a new log
I20260807 11:16:29.982627 5344 ts_tablet_manager.cc:1397] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 11:16:29.983644 5344 raft_consensus.cc:348] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.984009 5344 raft_consensus.cc:374] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.984146 5344 raft_consensus.cc:729] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303789002d94d4794f96859c10f8e44, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.984723 5344 consensus_queue.cc:260] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.985364 5351 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:29.985664 5344 ts_tablet_manager.cc:1428] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:29.985716 5351 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:29.987358 5346 tablet_bootstrap.cc:492] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: No bootstrap required, opened a new log
I20260807 11:16:29.987553 5346 ts_tablet_manager.cc:1397] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260807 11:16:29.988185 5346 raft_consensus.cc:348] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.988368 5346 raft_consensus.cc:374] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:29.988433 5346 raft_consensus.cc:729] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137a6b0550f436894a08df47d21c72d, State: Initialized, Role: FOLLOWER
I20260807 11:16:29.988663 5346 consensus_queue.cc:260] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:29.989328 5346 ts_tablet_manager.cc:1428] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 11:16:29.989148 5354 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:29.989573 5354 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:30.313977 5351 raft_consensus.cc:482] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:30.314236 5351 raft_consensus.cc:504] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.316058 5351 leader_election.cc:288] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:30.317798 5352 consensus_queue.cc:1035] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.325670 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.316632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48719 (local address 127.2.223.129:35489)
0807 11:16:30.316834 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.316841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.317503 (+ 662us) server_negotiation.cc:407] Connection header received
0807 11:16:30.317859 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.317864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.317926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.318031 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:30.318759 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.319909 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.322875 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.324037 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.324194 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.324268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.325202 (+ 934us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.325343 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:16:30.325708 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.316659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43583 (local address 127.2.223.131:41335)
0807 11:16:30.317321 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.317326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.317998 (+ 672us) server_negotiation.cc:407] Connection header received
0807 11:16:30.318302 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.318309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.318394 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.318530 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:30.319173 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.320113 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.322840 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.324011 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.324200 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.324268 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.325260 (+ 992us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.325405 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":576,"threads_started":1}
I20260807 11:16:30.326382 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.316404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41335 (local address 127.2.223.130:43583)
0807 11:16:30.317873 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:30.317893 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:30.318042 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:30.318583 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:30.318587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:30.318626 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:30.319005 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.319013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.320258 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.320263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.322655 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.322669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.324422 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.324429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.324609 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.324644 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:16:30.325873 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0807 11:16:30.325998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":30,"thread_start_us":1085,"threads_started":1}
I20260807 11:16:30.327158 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.316357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:35489 (local address 127.2.223.130:48719)
0807 11:16:30.317367 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:30.317396 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:30.317631 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:30.318103 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:30.318107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:30.318134 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:30.318583 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.318599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.320191 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.320200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.322699 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.322715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.324838 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.324844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.325007 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.325051 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 11:16:30.326388 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0807 11:16:30.326760 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":110,"thread_start_us":667,"threads_started":1}
I20260807 11:16:30.327234 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fcf74f49a4fcda5b7eb324c273b0a" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b" is_pre_election: true
I20260807 11:16:30.327575 5276 raft_consensus.cc:2454] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:30.327816 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fcf74f49a4fcda5b7eb324c273b0a" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d" is_pre_election: true
I20260807 11:16:30.328076 4984 raft_consensus.cc:2454] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:30.328140 5053 leader_election.cc:302] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:30.328532 5351 raft_consensus.cc:2781] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:30.328672 5351 raft_consensus.cc:482] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:30.328773 5351 raft_consensus.cc:3037] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.329113 5356 consensus_queue.cc:1035] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.332206 5351 raft_consensus.cc:504] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.333377 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fcf74f49a4fcda5b7eb324c273b0a" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d"
I20260807 11:16:30.333652 4984 raft_consensus.cc:3037] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.333565 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fcf74f49a4fcda5b7eb324c273b0a" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b"
I20260807 11:16:30.333837 5276 raft_consensus.cc:3037] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.336205 4984 raft_consensus.cc:2454] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:30.336608 5051 leader_election.cc:302] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5137a6b0550f436894a08df47d21c72d, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:30.336905 5375 raft_consensus.cc:2781] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:30.337769 5276 raft_consensus.cc:2454] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:30.338322 5351 leader_election.cc:288] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Requested vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:30.338841 5375 raft_consensus.cc:686] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [term 1 LEADER]: Becoming Leader. State: Replica: a303789002d94d4794f96859c10f8e44, State: Running, Role: LEADER
I20260807 11:16:30.339478 5375 consensus_queue.cc:237] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.341532 5351 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: TxnStatusTablet state changed. Reason: New leader a303789002d94d4794f96859c10f8e44. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.341765 5351 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:30.342535 5380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:16:30.342809 5380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:16:30.343966 4807 catalog_manager.cc:5495] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to a303789002d94d4794f96859c10f8e44 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } health_report { overall_health: UNKNOWN } } }
I20260807 11:16:30.438683 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.432154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.2.223.130:40939)
0807 11:16:30.432768 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.432781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.432799 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:16:30.432879 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.432882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.432953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.433149 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:30.433666 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.434470 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.436118 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.436576 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.436726 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.437279 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.437825 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.438022 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.438183 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.438266 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":282,"threads_started":1}
I20260807 11:16:30.438665 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.432100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40939 (local address 127.0.0.1:37006)
0807 11:16:30.432383 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:30.432410 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:30.432561 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:30.433163 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:30.433168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:30.433180 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:30.433483 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.433495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.434702 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.434706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.435965 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.435977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.436776 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.436784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.437019 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.437073 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.437946 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.438025 (+ 79us) client_negotiation.cc:769] Sending connection context
0807 11:16:30.438142 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:16:30.438332 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 11:16:30.446169 4984 raft_consensus.cc:1261] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:30.446444 5276 raft_consensus.cc:1261] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:30.446980 5375 consensus_queue.cc:1035] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.447304 5351 consensus_queue.cc:1035] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.454687 5352 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: TxnStatusTablet state changed. Reason: New leader a303789002d94d4794f96859c10f8e44. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.454978 5352 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:30.456602 5354 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: TxnStatusTablet state changed. Reason: New leader a303789002d94d4794f96859c10f8e44. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.456894 5354 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:30.458681 5377 mvcc.cc:205] Tried to move back new op lower bound from 7315871295258701824 to 7315871294834462720. Current Snapshot: MvccSnapshot[applied={T|T < 7315871295258701824}]
I20260807 11:16:30.460667 5352 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.460893 5352 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 2cb719d93353476bb464f6a78181b41b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:30.464601 5354 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.464855 5354 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P 5137a6b0550f436894a08df47d21c72d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:16:30.466461 5351 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.466800 5351 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:30.467980 5378 tablet_replica.cc:439] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } } }
I20260807 11:16:30.469527 5378 tablet_replica.cc:442] T 165fcf74f49a4fcda5b7eb324c273b0a P a303789002d94d4794f96859c10f8e44: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:16:30.481455 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.472129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.2.223.131:41335)
0807 11:16:30.472352 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.472357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.475176 (+ 2819us) server_negotiation.cc:407] Connection header received
0807 11:16:30.475438 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.475444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.475538 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.475727 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:30.476225 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.477354 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.478938 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.479588 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.479719 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.480129 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.480842 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.480999 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.481048 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.481146 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:16:30.498541 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.492443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:51931 (local address 127.2.223.130:40939)
0807 11:16:30.492701 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.492708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.493104 (+ 396us) server_negotiation.cc:407] Connection header received
0807 11:16:30.493317 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.493322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.493390 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.493462 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:30.493947 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.494758 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.496855 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.497251 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.497346 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.497875 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.497970 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.498037 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.498215 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.498277 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:16:30.498819 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.492391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40939 (local address 127.2.223.131:51931)
0807 11:16:30.492992 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:30.493013 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:30.493169 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:30.493510 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:30.493517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:30.493531 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:30.493799 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.493812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.494908 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.494914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.496619 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.496635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.497405 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.497411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.497589 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.497657 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.498070 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.498083 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 11:16:30.498184 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 11:16:30.498328 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":291,"threads_started":1}
I20260807 11:16:30.538130 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.531719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:55661 (local address 127.2.223.131:41335)
0807 11:16:30.531905 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:30.531913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:30.532589 (+ 676us) server_negotiation.cc:407] Connection header received
0807 11:16:30.532811 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:30.532816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:30.532916 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:30.533006 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:30.533726 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.534564 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.536100 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.536551 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.536707 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.537236 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.537393 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.537483 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:30.537744 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 11:16:30.537825 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:16:30.538372 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:30.531813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41335 (local address 127.2.223.130:55661)
0807 11:16:30.532473 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:30.532494 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:30.532644 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:30.533178 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:30.533184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:30.533196 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:30.533519 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.533536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.534768 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.534773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.535949 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:30.535962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:30.536752 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:30.536761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:30.536917 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:30.537008 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:30.537575 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:30.537585 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 11:16:30.537704 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 11:16:30.537856 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260807 11:16:30.581668 4807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:30.581979 4807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:30.586187 4807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Sending DeleteTablet for 3 replicas of tablet 6b3365ec87f848949a2e15a1806f47f0
I20260807 11:16:30.587508 4956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:30 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50292
I20260807 11:16:30.587504 5102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:30 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37002
I20260807 11:16:30.588997 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3365ec87f848949a2e15a1806f47f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:30 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:34820
I20260807 11:16:30.595229 4807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:30.595945 4807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:30.597422 5403 tablet_replica.cc:330] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: stopping tablet replica
I20260807 11:16:30.604175 5403 raft_consensus.cc:2229] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:30.604276 5101 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd (DEFAULT_TABLE table=test-table [id=30497d6e629e456995547dd8f2167bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:30.605014 5101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5c5c9b04124e56a4f173b4b310cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:30.605114 5255 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd (DEFAULT_TABLE table=test-table [id=30497d6e629e456995547dd8f2167bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:30.605607 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5c5c9b04124e56a4f173b4b310cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:30.606251 5403 raft_consensus.cc:2258] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:30.607116 4955 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd (DEFAULT_TABLE table=test-table [id=30497d6e629e456995547dd8f2167bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:30.607721 4955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5c5c9b04124e56a4f173b4b310cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:30.609344 5403 ts_tablet_manager.cc:1905] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:30.610831 5401 tablet_replica.cc:330] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: stopping tablet replica
I20260807 11:16:30.611346 5401 raft_consensus.cc:2229] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:30.616052 5401 raft_consensus.cc:2258] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:30.618074 5401 ts_tablet_manager.cc:1905] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:30.628320 5403 ts_tablet_manager.cc:1918] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:30.628799 5402 tablet_replica.cc:330] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: stopping tablet replica
I20260807 11:16:30.629205 5403 log.cc:1196] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/6b3365ec87f848949a2e15a1806f47f0
I20260807 11:16:30.629218 5402 raft_consensus.cc:2229] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:30.629590 5402 raft_consensus.cc:2258] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:30.629053 5404 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Bootstrap starting.
I20260807 11:16:30.631987 5402 ts_tablet_manager.cc:1905] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:30.633903 5401 ts_tablet_manager.cc:1918] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:30.634465 5403 ts_tablet_manager.cc:1939] T 6b3365ec87f848949a2e15a1806f47f0 P 2cb719d93353476bb464f6a78181b41b: Deleting consensus metadata
I20260807 11:16:30.635295 5405 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Bootstrap starting.
I20260807 11:16:30.634881 5404 tablet_bootstrap.cc:654] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:30.634119 5406 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Bootstrap starting.
I20260807 11:16:30.637795 5401 log.cc:1196] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/6b3365ec87f848949a2e15a1806f47f0
I20260807 11:16:30.639413 4792 catalog_manager.cc:4841] TS 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335): tablet 6b3365ec87f848949a2e15a1806f47f0 (table test-table [id=83da28ec70a7415496d9f0fe08f89c8a]) successfully deleted
I20260807 11:16:30.639999 5401 ts_tablet_manager.cc:1939] T 6b3365ec87f848949a2e15a1806f47f0 P 5137a6b0550f436894a08df47d21c72d: Deleting consensus metadata
I20260807 11:16:30.641975 4790 catalog_manager.cc:4841] TS 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489): tablet 6b3365ec87f848949a2e15a1806f47f0 (table test-table [id=83da28ec70a7415496d9f0fe08f89c8a]) successfully deleted
I20260807 11:16:30.642534 5402 ts_tablet_manager.cc:1918] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:30.642678 5402 log.cc:1196] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/6b3365ec87f848949a2e15a1806f47f0
I20260807 11:16:30.643111 5402 ts_tablet_manager.cc:1939] T 6b3365ec87f848949a2e15a1806f47f0 P a303789002d94d4794f96859c10f8e44: Deleting consensus metadata
I20260807 11:16:30.644459 5406 tablet_bootstrap.cc:654] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:30.644706 5404 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: No bootstrap required, opened a new log
I20260807 11:16:30.644871 4792 catalog_manager.cc:4841] TS a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939): tablet 6b3365ec87f848949a2e15a1806f47f0 (table test-table [id=83da28ec70a7415496d9f0fe08f89c8a]) successfully deleted
I20260807 11:16:30.644943 5404 ts_tablet_manager.cc:1397] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260807 11:16:30.646032 5404 raft_consensus.cc:348] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.646370 5405 tablet_bootstrap.cc:654] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:30.646391 5404 raft_consensus.cc:374] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:30.646531 5404 raft_consensus.cc:729] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Initialized, Role: FOLLOWER
I20260807 11:16:30.646880 5404 consensus_queue.cc:260] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.648641 5406 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: No bootstrap required, opened a new log
I20260807 11:16:30.649652 5406 ts_tablet_manager.cc:1397] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260807 11:16:30.650542 5404 ts_tablet_manager.cc:1428] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 11:16:30.650612 5406 raft_consensus.cc:348] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.650875 5406 raft_consensus.cc:374] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:30.650990 5406 raft_consensus.cc:729] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137a6b0550f436894a08df47d21c72d, State: Initialized, Role: FOLLOWER
I20260807 11:16:30.654704 5406 consensus_queue.cc:260] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.656353 5405 tablet_bootstrap.cc:492] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: No bootstrap required, opened a new log
I20260807 11:16:30.656525 5405 ts_tablet_manager.cc:1397] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260807 11:16:30.658638 5405 raft_consensus.cc:348] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.658970 5405 raft_consensus.cc:374] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:30.659158 5405 raft_consensus.cc:729] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303789002d94d4794f96859c10f8e44, State: Initialized, Role: FOLLOWER
I20260807 11:16:30.659556 5405 consensus_queue.cc:260] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.660462 5406 ts_tablet_manager.cc:1428] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260807 11:16:30.661540 5405 ts_tablet_manager.cc:1428] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:16:30.842808 5395 raft_consensus.cc:482] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:30.843111 5395 raft_consensus.cc:504] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.843952 5395 leader_election.cc:288] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939)
I20260807 11:16:30.844611 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5c5c9b04124e56a4f173b4b310cacd" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303789002d94d4794f96859c10f8e44" is_pre_election: true
I20260807 11:16:30.844611 4983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5c5c9b04124e56a4f173b4b310cacd" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d" is_pre_election: true
I20260807 11:16:30.844900 5125 raft_consensus.cc:2454] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 0.
I20260807 11:16:30.844926 4983 raft_consensus.cc:2454] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 0.
I20260807 11:16:30.845458 5198 leader_election.cc:302] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:30.845813 5395 raft_consensus.cc:2781] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:30.845969 5395 raft_consensus.cc:482] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:30.846103 5395 raft_consensus.cc:3037] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.849229 5395 raft_consensus.cc:504] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.850045 5395 leader_election.cc:288] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 election: Requested vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939)
I20260807 11:16:30.850768 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5c5c9b04124e56a4f173b4b310cacd" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303789002d94d4794f96859c10f8e44"
I20260807 11:16:30.851087 5125 raft_consensus.cc:3037] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.852432 4983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5c5c9b04124e56a4f173b4b310cacd" candidate_uuid: "2cb719d93353476bb464f6a78181b41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d"
I20260807 11:16:30.852622 4983 raft_consensus.cc:3037] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:30.854408 5125 raft_consensus.cc:2454] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 1.
I20260807 11:16:30.854869 5198 leader_election.cc:302] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:30.855262 5395 raft_consensus.cc:2781] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:30.855427 5395 raft_consensus.cc:686] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 1 LEADER]: Becoming Leader. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Running, Role: LEADER
I20260807 11:16:30.855748 5395 consensus_queue.cc:237] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:30.856704 4983 raft_consensus.cc:2454] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb719d93353476bb464f6a78181b41b in term 1.
I20260807 11:16:30.859513 4807 catalog_manager.cc:5495] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb719d93353476bb464f6a78181b41b (127.2.223.131). New cstate: current_term: 1 leader_uuid: "2cb719d93353476bb464f6a78181b41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } health_report { overall_health: HEALTHY } } }
I20260807 11:16:30.930035 4984 raft_consensus.cc:1261] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Refusing update from remote peer 2cb719d93353476bb464f6a78181b41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:30.930637 5125 raft_consensus.cc:1261] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Refusing update from remote peer 2cb719d93353476bb464f6a78181b41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:30.931674 5356 consensus_queue.cc:1035] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.932421 5395 consensus_queue.cc:1035] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:30.951630 5372 mvcc.cc:205] Tried to move back new op lower bound from 7315871297244323840 to 7315871296948432896. Current Snapshot: MvccSnapshot[applied={T|T < 7315871297244323840}]
W20260807 11:16:30.980684 5372 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 11:16:32.589130 4806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:32.589426 4806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:32.594835 4806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Sending DeleteTablet for 3 replicas of tablet 6a5c5c9b04124e56a4f173b4b310cacd
I20260807 11:16:32.595745 5102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:32 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37002
I20260807 11:16:32.595841 4955 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:32 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50292
I20260807 11:16:32.595914 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5c5c9b04124e56a4f173b4b310cacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:32 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:34820
I20260807 11:16:32.609089 5433 tablet_replica.cc:330] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: stopping tablet replica
I20260807 11:16:32.609298 5433 raft_consensus.cc:2229] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:32.609601 5433 raft_consensus.cc:2258] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:32.611636 5433 ts_tablet_manager.cc:1905] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:32.613236 5434 tablet_replica.cc:330] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: stopping tablet replica
I20260807 11:16:32.613610 5434 raft_consensus.cc:2229] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:32.614327 5434 raft_consensus.cc:2258] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:32.615765 5435 tablet_replica.cc:330] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: stopping tablet replica
I20260807 11:16:32.616058 5435 raft_consensus.cc:2229] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:32.616154 5434 ts_tablet_manager.cc:1905] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:32.616623 5435 raft_consensus.cc:2258] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:32.617707 5435 ts_tablet_manager.cc:1905] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:32.627480 5435 ts_tablet_manager.cc:1918] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:32.625391 5434 ts_tablet_manager.cc:1918] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:32.627754 5435 log.cc:1196] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/6a5c5c9b04124e56a4f173b4b310cacd
I20260807 11:16:32.628077 5434 log.cc:1196] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/6a5c5c9b04124e56a4f173b4b310cacd
I20260807 11:16:32.628473 5435 ts_tablet_manager.cc:1939] T 6a5c5c9b04124e56a4f173b4b310cacd P 5137a6b0550f436894a08df47d21c72d: Deleting consensus metadata
I20260807 11:16:32.628712 5434 ts_tablet_manager.cc:1939] T 6a5c5c9b04124e56a4f173b4b310cacd P 2cb719d93353476bb464f6a78181b41b: Deleting consensus metadata
I20260807 11:16:32.629624 5433 ts_tablet_manager.cc:1918] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:32.629793 5433 log.cc:1196] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/6a5c5c9b04124e56a4f173b4b310cacd
I20260807 11:16:32.630148 4792 catalog_manager.cc:4841] TS 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335): tablet 6a5c5c9b04124e56a4f173b4b310cacd (table test-table [id=30497d6e629e456995547dd8f2167bed]) successfully deleted
I20260807 11:16:32.630326 4790 catalog_manager.cc:4841] TS 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489): tablet 6a5c5c9b04124e56a4f173b4b310cacd (table test-table [id=30497d6e629e456995547dd8f2167bed]) successfully deleted
I20260807 11:16:32.630447 5433 ts_tablet_manager.cc:1939] T 6a5c5c9b04124e56a4f173b4b310cacd P a303789002d94d4794f96859c10f8e44: Deleting consensus metadata
I20260807 11:16:32.631704 4792 catalog_manager.cc:4841] TS a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939): tablet 6a5c5c9b04124e56a4f173b4b310cacd (table test-table [id=30497d6e629e456995547dd8f2167bed]) successfully deleted
Aug 07 11:16:32 dist-test-slave-5l3k krb5kdc[4759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:16:32 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:32.669505 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:32.652259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.2.223.190:35657)
0807 11:16:32.652908 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:32.652913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:32.652939 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:32.653118 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:32.653123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:32.653185 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:32.653319 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:32.654757 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.655575 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.657280 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.657696 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.657876 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:32.663172 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:32.663178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:32.663181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:32.663213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:32.666844 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:32.667460 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:32.667464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:32.667466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:32.667639 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:32.667951 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:32.667955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:32.667957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:32.668306 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:32.668567 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:32.668809 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 11:16:32.669013 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":336,"threads_started":1}
I20260807 11:16:32.673794 4806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:32.674671 4806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:32.681756 4955 tablet_service.cc:1467] Processing CreateTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f (DEFAULT_TABLE table=test-table [id=78adf9bfda334474a273011d43124640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:32.682228 4955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ced245e73e4bc2983dc3f7a471d16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:32.682199 5102 tablet_service.cc:1467] Processing CreateTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f (DEFAULT_TABLE table=test-table [id=78adf9bfda334474a273011d43124640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:32.682719 5102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ced245e73e4bc2983dc3f7a471d16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:32.688279 5445 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Bootstrap starting.
I20260807 11:16:32.689419 5255 tablet_service.cc:1467] Processing CreateTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f (DEFAULT_TABLE table=test-table [id=78adf9bfda334474a273011d43124640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:32.689929 5445 tablet_bootstrap.cc:654] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:32.690018 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ced245e73e4bc2983dc3f7a471d16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:32.691051 5446 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Bootstrap starting.
I20260807 11:16:32.692648 5446 tablet_bootstrap.cc:654] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:32.693447 5445 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: No bootstrap required, opened a new log
I20260807 11:16:32.693667 5445 ts_tablet_manager.cc:1397] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:32.694765 5445 raft_consensus.cc:348] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.695078 5445 raft_consensus.cc:374] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:32.695194 5445 raft_consensus.cc:729] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137a6b0550f436894a08df47d21c72d, State: Initialized, Role: FOLLOWER
I20260807 11:16:32.695544 5445 consensus_queue.cc:260] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.696970 5445 ts_tablet_manager.cc:1428] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:32.699115 5446 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: No bootstrap required, opened a new log
I20260807 11:16:32.699282 5446 ts_tablet_manager.cc:1397] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260807 11:16:32.700961 5452 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Bootstrap starting.
I20260807 11:16:32.700877 5446 raft_consensus.cc:348] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.701350 5446 raft_consensus.cc:374] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:32.701561 5446 raft_consensus.cc:729] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303789002d94d4794f96859c10f8e44, State: Initialized, Role: FOLLOWER
I20260807 11:16:32.701967 5446 consensus_queue.cc:260] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.702795 5446 ts_tablet_manager.cc:1428] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260807 11:16:32.704591 5452 tablet_bootstrap.cc:654] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:32.708637 5452 tablet_bootstrap.cc:492] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: No bootstrap required, opened a new log
I20260807 11:16:32.708832 5452 ts_tablet_manager.cc:1397] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260807 11:16:32.710100 5452 raft_consensus.cc:348] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.710405 5452 raft_consensus.cc:374] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:32.710664 5452 raft_consensus.cc:729] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Initialized, Role: FOLLOWER
I20260807 11:16:32.711041 5452 consensus_queue.cc:260] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.711721 5452 ts_tablet_manager.cc:1428] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 11:16:32.838670 5415 raft_consensus.cc:482] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:32.838891 5415 raft_consensus.cc:504] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.839788 5415 leader_election.cc:288] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:32.840476 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ced245e73e4bc2983dc3f7a471d16f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b" is_pre_election: true
I20260807 11:16:32.840420 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ced245e73e4bc2983dc3f7a471d16f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d" is_pre_election: true
I20260807 11:16:32.840777 4984 raft_consensus.cc:2454] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:32.840788 5276 raft_consensus.cc:2454] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:32.841352 5051 leader_election.cc:302] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, 5137a6b0550f436894a08df47d21c72d, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:32.841790 5415 raft_consensus.cc:2781] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:32.841997 5415 raft_consensus.cc:482] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:32.842188 5415 raft_consensus.cc:3037] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:32.845566 5415 raft_consensus.cc:504] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.846344 5415 leader_election.cc:288] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Requested vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:32.847039 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ced245e73e4bc2983dc3f7a471d16f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d"
I20260807 11:16:32.847048 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ced245e73e4bc2983dc3f7a471d16f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b"
I20260807 11:16:32.847347 4984 raft_consensus.cc:3037] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:32.847364 5276 raft_consensus.cc:3037] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:32.851356 5276 raft_consensus.cc:2454] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:32.851967 5053 leader_election.cc:302] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:32.852021 4984 raft_consensus.cc:2454] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:32.852602 5415 raft_consensus.cc:2781] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:32.852856 5415 raft_consensus.cc:686] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 1 LEADER]: Becoming Leader. State: Replica: a303789002d94d4794f96859c10f8e44, State: Running, Role: LEADER
I20260807 11:16:32.853399 5415 consensus_queue.cc:237] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:32.857736 4806 catalog_manager.cc:5495] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to a303789002d94d4794f96859c10f8e44 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } health_report { overall_health: UNKNOWN } } }
I20260807 11:16:32.905545 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:32.897559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.2.223.130:40939)
0807 11:16:32.898836 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:32.898844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:32.898892 (+ 48us) server_negotiation.cc:407] Connection header received
0807 11:16:32.898957 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:32.898965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:32.899050 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:32.899168 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:32.899842 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.901025 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.902944 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.903677 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.903779 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:32.904533 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:32.904696 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:32.904780 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:32.904931 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 11:16:32.904990 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":742,"threads_started":1}
I20260807 11:16:32.923584 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:32.916209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49861 (local address 127.2.223.130:40939)
0807 11:16:32.916373 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:32.916377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:32.917565 (+ 1188us) server_negotiation.cc:407] Connection header received
0807 11:16:32.917718 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:32.917730 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:32.917807 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:32.917882 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:32.918530 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.919554 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.921528 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.922146 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.922237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:32.922851 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:32.922952 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:32.923037 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:32.923172 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 11:16:32.923235 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:16:32.923655 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:32.916187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40939 (local address 127.2.223.130:49861)
0807 11:16:32.917474 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:32.917489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:32.917622 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:32.917910 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:32.917913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:32.917922 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:16:32.918363 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:32.918371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.919712 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.919715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:32.921399 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:32.921412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:32.922320 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:32.922325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:32.922643 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:32.922679 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:16:32.923057 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:16:32.923065 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 11:16:32.923127 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 11:16:32.923221 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":920,"threads_started":1}
I20260807 11:16:32.934139 5276 raft_consensus.cc:1261] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:32.934542 4984 raft_consensus.cc:1261] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:32.934832 5460 consensus_queue.cc:1035] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:32.935410 5351 consensus_queue.cc:1035] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:33.004781 4806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:33.005089 4806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:33.008929 4806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Sending DeleteTablet for 3 replicas of tablet f1ced245e73e4bc2983dc3f7a471d16f
I20260807 11:16:33.009956 4955 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:33 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50292
I20260807 11:16:33.010012 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:33 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:34820
I20260807 11:16:33.010507 5435 tablet_replica.cc:330] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: stopping tablet replica
I20260807 11:16:33.010526 5101 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ced245e73e4bc2983dc3f7a471d16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:33 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37002
I20260807 11:16:33.010969 5434 tablet_replica.cc:330] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: stopping tablet replica
I20260807 11:16:33.010900 5435 raft_consensus.cc:2229] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:33.011009 5433 tablet_replica.cc:330] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: stopping tablet replica
I20260807 11:16:33.011324 5434 raft_consensus.cc:2229] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:33.011330 5433 raft_consensus.cc:2229] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:33.011376 5435 raft_consensus.cc:2258] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:33.011735 5434 raft_consensus.cc:2258] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:33.011858 5433 raft_consensus.cc:2258] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:33.012722 4806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 11:16:33.013267 5433 ts_tablet_manager.cc:1905] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:33.013291 5435 ts_tablet_manager.cc:1905] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 11:16:33.013384 4806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:33.018955 5434 ts_tablet_manager.cc:1905] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:33.022529 5435 ts_tablet_manager.cc:1918] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:33.022725 5435 log.cc:1196] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/f1ced245e73e4bc2983dc3f7a471d16f
I20260807 11:16:33.023169 5435 ts_tablet_manager.cc:1939] T f1ced245e73e4bc2983dc3f7a471d16f P 5137a6b0550f436894a08df47d21c72d: Deleting consensus metadata
I20260807 11:16:33.025339 4790 catalog_manager.cc:4841] TS 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489): tablet f1ced245e73e4bc2983dc3f7a471d16f (table test-table [id=78adf9bfda334474a273011d43124640]) successfully deleted
I20260807 11:16:33.025730 4955 tablet_service.cc:1467] Processing CreateTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f (DEFAULT_TABLE table=test-table [id=3c881c727afa45b6b2c53a33c80767d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:33.026118 4955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160f5be7e6c4f61b0b750f5be16967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:33.026643 5255 tablet_service.cc:1467] Processing CreateTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f (DEFAULT_TABLE table=test-table [id=3c881c727afa45b6b2c53a33c80767d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:33.026962 5101 tablet_service.cc:1467] Processing CreateTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f (DEFAULT_TABLE table=test-table [id=3c881c727afa45b6b2c53a33c80767d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:33.027191 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160f5be7e6c4f61b0b750f5be16967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:33.027676 5101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160f5be7e6c4f61b0b750f5be16967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:33.035892 5445 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Bootstrap starting.
I20260807 11:16:33.036991 5452 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Bootstrap starting.
I20260807 11:16:33.039227 5445 tablet_bootstrap.cc:654] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:33.040596 5452 tablet_bootstrap.cc:654] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:33.046285 5433 ts_tablet_manager.cc:1918] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:33.048570 5445 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: No bootstrap required, opened a new log
I20260807 11:16:33.048856 5445 ts_tablet_manager.cc:1397] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260807 11:16:33.049917 5445 raft_consensus.cc:348] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.050374 5445 raft_consensus.cc:374] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:33.050585 5445 raft_consensus.cc:729] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137a6b0550f436894a08df47d21c72d, State: Initialized, Role: FOLLOWER
I20260807 11:16:33.046471 5434 ts_tablet_manager.cc:1918] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:16:33.051045 5434 log.cc:1196] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/f1ced245e73e4bc2983dc3f7a471d16f
I20260807 11:16:33.050952 5445 consensus_queue.cc:260] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.051532 5445 ts_tablet_manager.cc:1428] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260807 11:16:33.053206 5434 ts_tablet_manager.cc:1939] T f1ced245e73e4bc2983dc3f7a471d16f P 2cb719d93353476bb464f6a78181b41b: Deleting consensus metadata
I20260807 11:16:33.055579 4792 catalog_manager.cc:4841] TS 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335): tablet f1ced245e73e4bc2983dc3f7a471d16f (table test-table [id=78adf9bfda334474a273011d43124640]) successfully deleted
I20260807 11:16:33.046619 5446 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Bootstrap starting.
I20260807 11:16:33.058558 5446 tablet_bootstrap.cc:654] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:33.058732 5433 log.cc:1196] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/f1ced245e73e4bc2983dc3f7a471d16f
I20260807 11:16:33.059953 5433 ts_tablet_manager.cc:1939] T f1ced245e73e4bc2983dc3f7a471d16f P a303789002d94d4794f96859c10f8e44: Deleting consensus metadata
I20260807 11:16:33.060278 5452 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: No bootstrap required, opened a new log
I20260807 11:16:33.060631 5452 ts_tablet_manager.cc:1397] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.010s
I20260807 11:16:33.061488 4792 catalog_manager.cc:4841] TS a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939): tablet f1ced245e73e4bc2983dc3f7a471d16f (table test-table [id=78adf9bfda334474a273011d43124640]) successfully deleted
I20260807 11:16:33.062784 5452 raft_consensus.cc:348] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.063359 5452 raft_consensus.cc:374] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:33.063647 5452 raft_consensus.cc:729] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb719d93353476bb464f6a78181b41b, State: Initialized, Role: FOLLOWER
I20260807 11:16:33.064338 5446 tablet_bootstrap.cc:492] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: No bootstrap required, opened a new log
I20260807 11:16:33.064252 5452 consensus_queue.cc:260] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.064699 5446 ts_tablet_manager.cc:1397] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20260807 11:16:33.065368 5452 ts_tablet_manager.cc:1428] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:16:33.067876 5446 raft_consensus.cc:348] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.068292 5446 raft_consensus.cc:374] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:33.068534 5446 raft_consensus.cc:729] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303789002d94d4794f96859c10f8e44, State: Initialized, Role: FOLLOWER
I20260807 11:16:33.069070 5446 consensus_queue.cc:260] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.070070 5446 ts_tablet_manager.cc:1428] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:16:33.159518 5415 raft_consensus.cc:482] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:33.159946 5415 raft_consensus.cc:504] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.161001 5415 leader_election.cc:288] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:33.161464 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160f5be7e6c4f61b0b750f5be16967f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d" is_pre_election: true
I20260807 11:16:33.161785 4976 raft_consensus.cc:2454] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:33.162391 5051 leader_election.cc:302] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5137a6b0550f436894a08df47d21c72d, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:33.162339 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160f5be7e6c4f61b0b750f5be16967f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b" is_pre_election: true
I20260807 11:16:33.162817 5415 raft_consensus.cc:2781] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:33.162911 5415 raft_consensus.cc:482] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:33.162915 5275 raft_consensus.cc:2454] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 0.
I20260807 11:16:33.163009 5415 raft_consensus.cc:3037] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:33.166749 5415 raft_consensus.cc:504] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.168195 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160f5be7e6c4f61b0b750f5be16967f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137a6b0550f436894a08df47d21c72d"
I20260807 11:16:33.168380 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160f5be7e6c4f61b0b750f5be16967f" candidate_uuid: "a303789002d94d4794f96859c10f8e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb719d93353476bb464f6a78181b41b"
I20260807 11:16:33.168540 4976 raft_consensus.cc:3037] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:33.168646 5275 raft_consensus.cc:3037] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:33.173134 5275 raft_consensus.cc:2454] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:33.173753 5053 leader_election.cc:302] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb719d93353476bb464f6a78181b41b, a303789002d94d4794f96859c10f8e44; no voters:
I20260807 11:16:33.174103 5460 raft_consensus.cc:2781] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:33.174320 5415 leader_election.cc:288] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [CANDIDATE]: Term 1 election: Requested vote from peers 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489), 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335)
I20260807 11:16:33.174453 5460 raft_consensus.cc:686] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 1 LEADER]: Becoming Leader. State: Replica: a303789002d94d4794f96859c10f8e44, State: Running, Role: LEADER
I20260807 11:16:33.175017 5460 consensus_queue.cc:237] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } }
I20260807 11:16:33.178067 4976 raft_consensus.cc:2454] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303789002d94d4794f96859c10f8e44 in term 1.
I20260807 11:16:33.182451 4806 catalog_manager.cc:5495] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to a303789002d94d4794f96859c10f8e44 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "a303789002d94d4794f96859c10f8e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a303789002d94d4794f96859c10f8e44" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 } health_report { overall_health: UNKNOWN } } }
I20260807 11:16:33.229770 4984 raft_consensus.cc:1261] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:33.230944 5415 consensus_queue.cc:1035] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137a6b0550f436894a08df47d21c72d" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:33.235497 5275 raft_consensus.cc:1261] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Refusing update from remote peer a303789002d94d4794f96859c10f8e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:33.236428 5351 consensus_queue.cc:1035] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb719d93353476bb464f6a78181b41b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:16:33.276283 5413 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 11:16:34.887290 4807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:34.887624 4807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:34.892146 4807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79cc3da98fc74d36b42f79836ab86378: Sending DeleteTablet for 3 replicas of tablet 5160f5be7e6c4f61b0b750f5be16967f
I20260807 11:16:34.892956 5102 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:34 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37002
I20260807 11:16:34.893042 4955 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:34 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50292
I20260807 11:16:34.893296 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160f5be7e6c4f61b0b750f5be16967f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:34 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:34820
I20260807 11:16:34.893911 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4886
I20260807 11:16:34.897914 5495 tablet_replica.cc:330] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: stopping tablet replica
I20260807 11:16:34.898262 5495 raft_consensus.cc:2229] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:34.898849 5495 raft_consensus.cc:2258] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:34.900221 5495 ts_tablet_manager.cc:1905] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:34.905920 5496 tablet_replica.cc:330] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: stopping tablet replica
I20260807 11:16:34.906035 5494 tablet_replica.cc:330] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: stopping tablet replica
I20260807 11:16:34.906208 5496 raft_consensus.cc:2229] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:34.906369 5494 raft_consensus.cc:2229] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:34.906560 5496 raft_consensus.cc:2258] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:34.906785 5494 raft_consensus.cc:2258] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:34.907732 5496 ts_tablet_manager.cc:1905] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:34.907995 5494 ts_tablet_manager.cc:1905] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:34.912308 5495 ts_tablet_manager.cc:1918] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:34.912560 5495 log.cc:1196] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/5160f5be7e6c4f61b0b750f5be16967f
I20260807 11:16:34.913235 5495 ts_tablet_manager.cc:1939] T 5160f5be7e6c4f61b0b750f5be16967f P a303789002d94d4794f96859c10f8e44: Deleting consensus metadata
I20260807 11:16:34.914441 4792 catalog_manager.cc:4841] TS a303789002d94d4794f96859c10f8e44 (127.2.223.130:40939): tablet 5160f5be7e6c4f61b0b750f5be16967f (table test-table [id=3c881c727afa45b6b2c53a33c80767d2]) successfully deleted
I20260807 11:16:34.915337 5496 ts_tablet_manager.cc:1918] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:34.915545 5496 log.cc:1196] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/5160f5be7e6c4f61b0b750f5be16967f
I20260807 11:16:34.916230 5494 ts_tablet_manager.cc:1918] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:16:34.916342 5496 ts_tablet_manager.cc:1939] T 5160f5be7e6c4f61b0b750f5be16967f P 2cb719d93353476bb464f6a78181b41b: Deleting consensus metadata
I20260807 11:16:34.916477 5494 log.cc:1196] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/5160f5be7e6c4f61b0b750f5be16967f
I20260807 11:16:34.917196 5494 ts_tablet_manager.cc:1939] T 5160f5be7e6c4f61b0b750f5be16967f P 5137a6b0550f436894a08df47d21c72d: Deleting consensus metadata
I20260807 11:16:34.917582 4792 catalog_manager.cc:4841] TS 2cb719d93353476bb464f6a78181b41b (127.2.223.131:41335): tablet 5160f5be7e6c4f61b0b750f5be16967f (table test-table [id=3c881c727afa45b6b2c53a33c80767d2]) successfully deleted
I20260807 11:16:34.918242 4790 catalog_manager.cc:4841] TS 5137a6b0550f436894a08df47d21c72d (127.2.223.129:35489): tablet 5160f5be7e6c4f61b0b750f5be16967f (table test-table [id=3c881c727afa45b6b2c53a33c80767d2]) successfully deleted
Aug 07 11:16:34 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:34.931483 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:34.895399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.2.223.129:35489)
0807 11:16:34.904027 (+ 8628us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:34.904036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:34.906570 (+ 2534us) server_negotiation.cc:407] Connection header received
0807 11:16:34.906893 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:34.906898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:34.906977 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:34.907116 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:34.908980 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:34.910255 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:34.912375 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:34.913293 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:34.913450 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:34.924453 (+ 11003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:34.924469 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:34.924486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:34.924518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:34.928674 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:34.929408 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:34.929416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:34.929418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:34.929586 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:34.929950 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:34.929953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:34.929955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:34.930335 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:34.930685 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:34.930909 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:16:34.931071 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8418,"thread_start_us":8199,"threads_started":1}
W20260807 11:16:34.932405 5024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50294
W20260807 11:16:34.933336 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:34.933633 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4886
I20260807 11:16:34.963989 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5034
Aug 07 11:16:34 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:34.980319 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:34.964805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.2.223.130:40939)
0807 11:16:34.965402 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:34.965409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:34.965430 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:16:34.965499 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:34.965504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:34.965581 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:34.965695 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:34.967046 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:34.968226 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:34.969796 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:34.970245 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:34.970399 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:34.974645 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:34.974659 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:34.974675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:34.974740 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:34.977538 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:34.978244 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:34.978251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:34.978253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:34.978381 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:34.978764 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:34.978770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:34.978773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:34.979132 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:34.979456 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:34.979738 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 11:16:34.979979 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
W20260807 11:16:34.980942 5171 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37022
W20260807 11:16:34.981530 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:34.981657 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5034
I20260807 11:16:35.015367 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5180
Aug 07 11:16:35 dist-test-slave-5l3k krb5kdc[4759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:35.033918 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:35.016277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.2.223.131:41335)
0807 11:16:35.016918 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:35.016922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:35.016937 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:16:35.016992 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:35.016995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:35.017063 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:35.017149 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:35.019312 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:35.020232 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:35.022463 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:35.023260 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:35.023381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:35.027821 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:35.027839 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:35.027857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:35.027904 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:35.031335 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:35.032009 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:35.032017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:35.032019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:35.032163 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:35.032480 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:35.032484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:35.032485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:35.032849 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:35.033119 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:35.033288 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 11:16:35.033455 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":397,"threads_started":1}
W20260807 11:16:35.034693 5318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34838
W20260807 11:16:35.035337 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5180: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:35.035521 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5180
I20260807 11:16:35.063153 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 4774
I20260807 11:16:35.064443 4853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:35.185074 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 4774
2026-08-07T11:16:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10595 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:35 dist-test-slave-5l3k krb5kdc[5504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:35 dist-test-slave-5l3k krb5kdc[5504](info): set up 2 sockets
Aug 07 11:16:35 dist-test-slave-5l3k krb5kdc[5504](info): commencing operation
krb5kdc: starting...
W20260807 11:16:36.310937 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:36 dist-test-slave-5l3k krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:36.522423 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40853
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:36.852793 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:36.853189 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:36.853376 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:36.864449 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:36.864552 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:36.864607 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:36.864666 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:36.864717 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:36.874734 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:36.875991 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:36.877278 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:36.889711 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:36.889724 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:36.889748 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:36.895741 5518 server_base.cc:1034] running on GCE node
I20260807 11:16:36.896554 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:36.898068 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:36.899307 5518 hybrid_clock.cc:648] HybridClock initialized: now 1786101396899269 us; error 53 us; skew 500 ppm
I20260807 11:16:36.899669 5518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:36 dist-test-slave-5l3k krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:36.902894 5518 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:36.904404 5518 webserver.cc:460] Webserver started at http://127.2.223.190:34917/ using document root <none> and password file <none>
I20260807 11:16:36.905126 5518 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:36.905272 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:36.905591 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:36.908510 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf"
format_stamp: "Formatted at 2026-08-07 11:16:36 on dist-test-slave-5l3k"
server_key: "19b695ea03fe2de0c8c2bf0ba99c2895"
server_key_iv: "27a51aeeeedbcad8b9141bed155b583a"
server_key_version: "encryptionkey@0"
I20260807 11:16:36.909294 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf"
format_stamp: "Formatted at 2026-08-07 11:16:36 on dist-test-slave-5l3k"
server_key: "19b695ea03fe2de0c8c2bf0ba99c2895"
server_key_iv: "27a51aeeeedbcad8b9141bed155b583a"
server_key_version: "encryptionkey@0"
I20260807 11:16:36.914414 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:16:36.917593 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:36.918901 5518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 11:16:36.919091 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf"
format_stamp: "Formatted at 2026-08-07 11:16:36 on dist-test-slave-5l3k"
server_key: "19b695ea03fe2de0c8c2bf0ba99c2895"
server_key_iv: "27a51aeeeedbcad8b9141bed155b583a"
server_key_version: "encryptionkey@0"
I20260807 11:16:36.919275 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:36.947376 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:36.951687 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:36.952001 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:36.971794 5518 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:40853
I20260807 11:16:36.971823 5588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:40853 every 8 connection(s)
I20260807 11:16:36.973256 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:36.976939 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:36.983130 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5518
I20260807 11:16:36.983402 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:36.984272 2942 external_mini_cluster.cc:1428] Setting key 339cbfc029d407cae2e8952183b602bf
I20260807 11:16:36.988235 5589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf: Bootstrap starting.
I20260807 11:16:36.991397 5589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:36.992807 5589 log.cc:824] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:36.995972 5589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf: No bootstrap required, opened a new log
Aug 07 11:16:36 dist-test-slave-5l3k krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:37.004514 5589 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } }
I20260807 11:16:37.004998 5589 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:37.005112 5589 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b6adc3f7bf4d728e02b5bd9dad7ddf, State: Initialized, Role: FOLLOWER
I20260807 11:16:37.005749 5589 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } }
I20260807 11:16:37.005992 5589 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:37.006191 5589 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:37.006397 5589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:37.008381 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:36.986441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.2.223.190:40853)
0807 11:16:36.987269 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:36.987281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:36.987322 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:16:36.989197 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:36.989226 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:36.989626 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:36.990003 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:36.991662 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:36.993676 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:36.995252 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:36.996267 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:36.996473 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:37.000789 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:37.000797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:37.000812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:37.000851 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:37.004011 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:37.004852 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:37.004858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:37.004864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:37.005142 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:37.005550 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:37.005555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:37.005558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:37.006205 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:37.006558 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:37.006956 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 11:16:37.007352 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":370,"threads_started":1}
I20260807 11:16:37.009047 5589 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } }
I20260807 11:16:37.009543 5589 leader_election.cc:302] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b6adc3f7bf4d728e02b5bd9dad7ddf; no voters:
I20260807 11:16:37.010207 5589 leader_election.cc:288] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:37.010329 5594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:37.011794 5594 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [term 1 LEADER]: Becoming Leader. State: Replica: a3b6adc3f7bf4d728e02b5bd9dad7ddf, State: Running, Role: LEADER
I20260807 11:16:37.012377 5594 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } }
I20260807 11:16:37.012703 5589 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:37.016013 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b6adc3f7bf4d728e02b5bd9dad7ddf. Latest consensus state: current_term: 1 leader_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } } }
I20260807 11:16:37.016121 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b6adc3f7bf4d728e02b5bd9dad7ddf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40853 } } }
I20260807 11:16:37.016541 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:37.016548 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:37.021754 5601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:37.027170 5601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:37.035619 5601 catalog_manager.cc:1347] Generated new cluster ID: 26424cfaa83840a48b872c641cf5a9db
I20260807 11:16:37.035753 5601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:37.052736 5601 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:37.053925 5601 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:37.061415 5601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3b6adc3f7bf4d728e02b5bd9dad7ddf: Generated new TSK 0
I20260807 11:16:37.062161 5601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:37.122566 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:37.456023 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:37.456284 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:37.456430 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:37.463349 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:37.463478 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:37.463655 5616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:37.473111 5616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:37.474459 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:37.475843 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:37.488758 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:37.489044 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:37.489073 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:37.490177 5616 server_base.cc:1034] running on GCE node
I20260807 11:16:37.490903 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:37.491858 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:37.493064 5616 hybrid_clock.cc:648] HybridClock initialized: now 1786101397493012 us; error 40 us; skew 500 ppm
I20260807 11:16:37.493348 5616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:37 dist-test-slave-5l3k krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:37.496480 5616 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:37.497925 5616 webserver.cc:460] Webserver started at http://127.2.223.129:32805/ using document root <none> and password file <none>
I20260807 11:16:37.498683 5616 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:37.498818 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:37.499217 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:37.502156 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "e53c334cbf1c44ad86d0656705a63861"
format_stamp: "Formatted at 2026-08-07 11:16:37 on dist-test-slave-5l3k"
server_key: "2e379ce4fefa7265536aee24a0f754a0"
server_key_iv: "3c84e103a50840f9b5e45a08506cbbeb"
server_key_version: "encryptionkey@0"
I20260807 11:16:37.502974 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "e53c334cbf1c44ad86d0656705a63861"
format_stamp: "Formatted at 2026-08-07 11:16:37 on dist-test-slave-5l3k"
server_key: "2e379ce4fefa7265536aee24a0f754a0"
server_key_iv: "3c84e103a50840f9b5e45a08506cbbeb"
server_key_version: "encryptionkey@0"
I20260807 11:16:37.508271 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:16:37.511791 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:37.513149 5616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:37.513324 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "e53c334cbf1c44ad86d0656705a63861"
format_stamp: "Formatted at 2026-08-07 11:16:37 on dist-test-slave-5l3k"
server_key: "2e379ce4fefa7265536aee24a0f754a0"
server_key_iv: "3c84e103a50840f9b5e45a08506cbbeb"
server_key_version: "encryptionkey@0"
I20260807 11:16:37.513533 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:37.544349 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:37.548267 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:37.548556 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:37.549515 5616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:37.550917 5616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:37.550992 5616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:37.551194 5616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:37.551256 5616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:37.585355 5616 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:44489
I20260807 11:16:37.585404 5753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:44489 every 8 connection(s)
I20260807 11:16:37.586745 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:37.596462 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5616
I20260807 11:16:37.596714 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:37.597628 2942 external_mini_cluster.cc:1428] Setting key 041db6ced4d0584f7940c40e8add7e8a
Aug 07 11:16:37 dist-test-slave-5l3k krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:37.608184 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:37.589456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:49179 (local address 127.2.223.190:40853)
0807 11:16:37.589948 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:37.589952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:37.590987 (+ 1035us) server_negotiation.cc:407] Connection header received
0807 11:16:37.592841 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:37.592846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:37.592920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:37.593074 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:37.594832 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:37.596039 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:37.598224 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:37.598804 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:37.598948 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:37.602537 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:37.602545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:37.602549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:37.602592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:37.605147 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:37.605850 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:37.605857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:37.605861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:37.605987 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:37.606430 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:37.606437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:37.606440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:37.606739 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:37.606895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:37.607593 (+ 698us) server_negotiation.cc:299] Negotiation successful
0807 11:16:37.607823 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260807 11:16:37.609143 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:37.590025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40853 (local address 127.2.223.129:49179)
0807 11:16:37.590804 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:37.590843 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:37.592568 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:37.593276 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:37.593284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:37.593813 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:37.594608 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:37.594639 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:37.596271 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:37.596276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:37.598073 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:37.598086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:37.599055 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:37.599065 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:37.599382 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:37.600124 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:37.600145 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:37.602295 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:37.605287 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:37.605296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:37.605312 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:37.605728 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:37.606115 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:37.606120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:37.606123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:37.606314 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:37.606871 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:37.606878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:37.607207 (+ 329us) client_negotiation.cc:769] Sending connection context
0807 11:16:37.607477 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 11:16:37.607971 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":294,"threads_started":1}
I20260807 11:16:37.610952 5754 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40853
I20260807 11:16:37.611271 5754 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:37.612165 5754 heartbeater.cc:511] Master 127.2.223.190:40853 requested a full tablet report, sending...
I20260807 11:16:37.614718 5552 ts_manager.cc:194] Registered new tserver with Master: e53c334cbf1c44ad86d0656705a63861 (127.2.223.129:44489)
I20260807 11:16:37.617097 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:49179
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:37.647898 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:37.973598 5760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:37.973865 5760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:37.974025 5760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:37.982254 5760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:37.982365 5760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:37.982573 5760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:37.992475 5760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:37.993728 5760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:37.995221 5760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:38.008112 5769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:38.008172 5770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:38.008071 5772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:38.009295 5760 server_base.cc:1034] running on GCE node
I20260807 11:16:38.009918 5760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:38.010895 5760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:38.012144 5760 hybrid_clock.cc:648] HybridClock initialized: now 1786101398012083 us; error 56 us; skew 500 ppm
I20260807 11:16:38.012408 5760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:38.015679 5760 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:38.017092 5760 webserver.cc:460] Webserver started at http://127.2.223.130:37849/ using document root <none> and password file <none>
I20260807 11:16:38.017782 5760 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:38.017912 5760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:38.018278 5760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:38.021087 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "42d1b6d0d5a24b95a02dd0282a20ba8e"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "0e3851a3f0ed7fe5bc3b0b41d1657748"
server_key_iv: "7c61b2f716865a5bb516f590d159083e"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.021957 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "42d1b6d0d5a24b95a02dd0282a20ba8e"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "0e3851a3f0ed7fe5bc3b0b41d1657748"
server_key_iv: "7c61b2f716865a5bb516f590d159083e"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.027163 5760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:16:38.030521 5780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.031780 5760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:16:38.031911 5760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "42d1b6d0d5a24b95a02dd0282a20ba8e"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "0e3851a3f0ed7fe5bc3b0b41d1657748"
server_key_iv: "7c61b2f716865a5bb516f590d159083e"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.032085 5760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:38.055150 5760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:38.059548 5760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:38.059795 5760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:38.060652 5760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:38.061950 5760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:38.062022 5760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.062103 5760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:38.062145 5760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.095304 5760 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:38795
I20260807 11:16:38.095378 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:38795 every 8 connection(s)
I20260807 11:16:38.096563 5760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:38.097122 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5760
I20260807 11:16:38.097363 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:38.097975 2942 external_mini_cluster.cc:1428] Setting key 24127b89dac755cf9611216bfb4f5d62
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:38.120960 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:38.099411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43123 (local address 127.2.223.190:40853)
0807 11:16:38.099590 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:38.099595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:38.101051 (+ 1456us) server_negotiation.cc:407] Connection header received
0807 11:16:38.104588 (+ 3537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:38.104592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:38.104659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:38.104738 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:38.107037 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.107733 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.110224 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.110743 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.110889 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:38.114935 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:38.114939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:38.114943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:38.114993 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:38.117842 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:38.118589 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:38.118595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:38.118598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:38.118731 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:38.119160 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:38.119164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:38.119167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:38.119481 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:38.119621 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:38.120525 (+ 904us) server_negotiation.cc:299] Negotiation successful
0807 11:16:38.120698 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 11:16:38.122681 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:38.100009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40853 (local address 127.2.223.130:43123)
0807 11:16:38.100864 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:38.100916 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:38.104251 (+ 3335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:38.105093 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:38.105108 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:38.105924 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:38.106857 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:38.106885 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.107893 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.107901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:38.110059 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:38.110077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.110980 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.110988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:38.111221 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:38.111925 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:38.111959 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:38.114716 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:38.117972 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:38.117981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:38.117997 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:38.118430 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:38.118844 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:38.118849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:38.118852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:38.119035 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:38.119663 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:38.119672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:38.120109 (+ 437us) client_negotiation.cc:769] Sending connection context
0807 11:16:38.120445 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 11:16:38.121089 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":310,"threads_started":1}
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:16:38.124363 5899 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40853
I20260807 11:16:38.124712 5899 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:38.125432 5899 heartbeater.cc:511] Master 127.2.223.190:40853 requested a full tablet report, sending...
I20260807 11:16:38.126906 5552 ts_manager.cc:194] Registered new tserver with Master: 42d1b6d0d5a24b95a02dd0282a20ba8e (127.2.223.130:38795)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
I20260807 11:16:38.127835 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:43123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:38.153589 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:38.480985 5905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:38.481302 5905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:38.481482 5905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:38.488476 5905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:38.488628 5905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:38.488802 5905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:38.499051 5905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:38.500313 5905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:38.501734 5905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:38.515014 5911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:38.515120 5914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:38.515101 5910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:38.516268 5905 server_base.cc:1034] running on GCE node
I20260807 11:16:38.516876 5905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:38.517910 5905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:38.519105 5905 hybrid_clock.cc:648] HybridClock initialized: now 1786101398519071 us; error 31 us; skew 500 ppm
I20260807 11:16:38.519371 5905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:38.522632 5905 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:38.524176 5905 webserver.cc:460] Webserver started at http://127.2.223.131:34057/ using document root <none> and password file <none>
I20260807 11:16:38.524791 5905 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:38.524890 5905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:38.525144 5905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:38.527963 5905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "3ff3035d8a9e42f7b22d931f31320942"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "6b4753d4937a409cd9a5a3ab5cd79a8b"
server_key_iv: "c910384126490e50324fd3c61185b149"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.528715 5905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff3035d8a9e42f7b22d931f31320942"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "6b4753d4937a409cd9a5a3ab5cd79a8b"
server_key_iv: "c910384126490e50324fd3c61185b149"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.533610 5905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:16:38.537088 5923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.538308 5905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:38.538513 5905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "3ff3035d8a9e42f7b22d931f31320942"
format_stamp: "Formatted at 2026-08-07 11:16:38 on dist-test-slave-5l3k"
server_key: "6b4753d4937a409cd9a5a3ab5cd79a8b"
server_key_iv: "c910384126490e50324fd3c61185b149"
server_key_version: "encryptionkey@0"
I20260807 11:16:38.538738 5905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:38.565393 5905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:38.569262 5905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:38.569587 5905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:38.570550 5905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:38.571884 5905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:38.572013 5905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.572173 5905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:38.572252 5905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:38.609095 5905 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:41429
I20260807 11:16:38.609143 6039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:41429 every 8 connection(s)
I20260807 11:16:38.610383 5905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:38.614662 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 5905
I20260807 11:16:38.614859 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:38.615429 2942 external_mini_cluster.cc:1428] Setting key 416d79feb9506ab6f38f898176fdb0a1
I20260807 11:16:38.620903 5754 heartbeater.cc:503] Master 127.2.223.190:40853 was elected leader, sending a full tablet report...
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:38.632604 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:38.612967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:58181 (local address 127.2.223.190:40853)
0807 11:16:38.613180 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:38.613188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:38.614705 (+ 1517us) server_negotiation.cc:407] Connection header received
0807 11:16:38.617099 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:38.617104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:38.617188 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:38.617309 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:38.619169 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.619934 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.621818 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.622377 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.622535 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:38.626276 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:38.626282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:38.626285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:38.626316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:38.629242 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:38.630212 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:38.630218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:38.630221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:38.630410 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:38.630968 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:38.630975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:38.630978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:38.631339 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:38.631454 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:38.632198 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 11:16:38.632362 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:16:38.633498 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:38.613496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40853 (local address 127.2.223.131:58181)
0807 11:16:38.614503 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:38.614575 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:38.616721 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:38.617578 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:38.617590 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:38.618354 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:38.618997 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:38.619014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.620091 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.620097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:38.621668 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:38.621683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:38.622887 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:38.622899 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:38.623113 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:38.623854 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:38.623889 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:38.626077 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:38.629387 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:38.629398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:38.629417 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:38.630019 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:38.630573 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:38.630580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:38.630587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:38.630790 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:38.631459 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:38.631464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:38.631749 (+ 285us) client_negotiation.cc:769] Sending connection context
0807 11:16:38.632048 (+ 299us) client_negotiation.cc:240] Negotiation successful
0807 11:16:38.632419 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":482,"threads_started":1}
I20260807 11:16:38.634867 6040 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40853
I20260807 11:16:38.635182 6040 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:38.635916 6040 heartbeater.cc:511] Master 127.2.223.190:40853 requested a full tablet report, sending...
I20260807 11:16:38.637332 5552 ts_manager.cc:194] Registered new tserver with Master: 3ff3035d8a9e42f7b22d931f31320942 (127.2.223.131:41429)
I20260807 11:16:38.638135 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:58181
I20260807 11:16:38.643255 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:16:38.680716 5756 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:16:38.677810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.2.223.190:40853)
0807 11:16:38.677990 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:38.677995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:38.678590 (+ 595us) server_negotiation.cc:407] Connection header received
0807 11:16:38.678745 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:38.678749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:38.678828 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:38.678923 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:38.680412 (+ 1489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57832: BlockingRecv error: recv got EOF from 127.0.0.1:57832 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:16:38.683887 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5616
W20260807 11:16:38.686673 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5616: Not authorized: Client connection negotiation failed: client connection to 127.2.223.129:44489: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:16:38.686875 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5616
I20260807 11:16:38.707460 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5760
W20260807 11:16:38.710237 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5760: Not authorized: Client connection negotiation failed: client connection to 127.2.223.130:38795: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:16:38.710402 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5760
I20260807 11:16:38.730952 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5905
W20260807 11:16:38.734062 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5905: Not authorized: Client connection negotiation failed: client connection to 127.2.223.131:41429: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:16:38.734222 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5905
I20260807 11:16:38.753189 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 5518
I20260807 11:16:38.754436 5584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:38.863818 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 5518
2026-08-07T11:16:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3674 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[6058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[6058](info): set up 2 sockets
Aug 07 11:16:38 dist-test-slave-5l3k krb5kdc[6058](info): commencing operation
krb5kdc: starting...
W20260807 11:16:39.991391 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:40.210137 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33825
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:40.543911 6072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:40.544241 6072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:40.544409 6072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:40.551570 6072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:40.551703 6072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:40.551817 6072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:40.551926 6072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:40.552009 6072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:40.561658 6072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:40.562911 6072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:40.564253 6072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:40.576229 6077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:40.576303 6078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:40.576378 6080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:40.576970 6072 server_base.cc:1034] running on GCE node
I20260807 11:16:40.577776 6072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:40.579154 6072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:40.580386 6072 hybrid_clock.cc:648] HybridClock initialized: now 1786101400580331 us; error 42 us; skew 500 ppm
I20260807 11:16:40.580729 6072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:40.587782 6072 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:40.589290 6072 webserver.cc:460] Webserver started at http://127.2.223.190:39801/ using document root <none> and password file <none>
I20260807 11:16:40.590034 6072 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:40.590178 6072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:40.590536 6072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:40.593513 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "1214382d30ae4af898176d59577be152"
format_stamp: "Formatted at 2026-08-07 11:16:40 on dist-test-slave-5l3k"
server_key: "8a904d8440260add5370342552f1331b"
server_key_iv: "25d425a509ae3fa79a3a855b9ef2d94b"
server_key_version: "encryptionkey@0"
I20260807 11:16:40.594309 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "1214382d30ae4af898176d59577be152"
format_stamp: "Formatted at 2026-08-07 11:16:40 on dist-test-slave-5l3k"
server_key: "8a904d8440260add5370342552f1331b"
server_key_iv: "25d425a509ae3fa79a3a855b9ef2d94b"
server_key_version: "encryptionkey@0"
I20260807 11:16:40.599452 6072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:16:40.602893 6088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:40.604208 6072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:16:40.604406 6072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "1214382d30ae4af898176d59577be152"
format_stamp: "Formatted at 2026-08-07 11:16:40 on dist-test-slave-5l3k"
server_key: "8a904d8440260add5370342552f1331b"
server_key_iv: "25d425a509ae3fa79a3a855b9ef2d94b"
server_key_version: "encryptionkey@0"
I20260807 11:16:40.604602 6072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:40.622248 6072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:40.626042 6072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:40.626331 6072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:40.645967 6072 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33825
I20260807 11:16:40.645991 6141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33825 every 8 connection(s)
I20260807 11:16:40.647349 6072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:40.650944 6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:40.651005 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6072
I20260807 11:16:40.651216 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:40.651876 2942 external_mini_cluster.cc:1428] Setting key a0ba67ae6a0c20f7795a1e0f78db1931
I20260807 11:16:40.665547 6142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152: Bootstrap starting.
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:40.669039 6142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:40.670452 6142 log.cc:824] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:40.673838 6142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152: No bootstrap required, opened a new log
I20260807 11:16:40.677452 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:40.653640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.2.223.190:33825)
0807 11:16:40.654549 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:40.654564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:40.654616 (+ 52us) server_negotiation.cc:407] Connection header received
0807 11:16:40.657549 (+ 2933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:40.657580 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:40.658150 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:40.658645 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:40.660005 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:40.662026 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:40.663381 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:40.664413 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:40.664588 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:40.668779 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:40.668792 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:40.668814 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:40.668869 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:40.672790 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:40.673522 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:40.673530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:40.673537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:40.673757 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:40.674156 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:40.674161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:40.674164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:40.674948 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:40.675275 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:40.675576 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 11:16:40.676064 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":384,"threads_started":1}
I20260807 11:16:40.681193 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:16:40.682902 6142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } }
I20260807 11:16:40.683264 6142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:40.683372 6142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1214382d30ae4af898176d59577be152, State: Initialized, Role: FOLLOWER
I20260807 11:16:40.683923 6142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } }
I20260807 11:16:40.684152 6142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:40.684297 6142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:40.684463 6142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:40.686688 6142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } }
I20260807 11:16:40.687175 6142 leader_election.cc:302] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1214382d30ae4af898176d59577be152; no voters:
I20260807 11:16:40.687778 6142 leader_election.cc:288] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:40.687934 6147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:40.689805 6142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:40.690776 6147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [term 1 LEADER]: Becoming Leader. State: Replica: 1214382d30ae4af898176d59577be152, State: Running, Role: LEADER
I20260807 11:16:40.691458 6147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } }
I20260807 11:16:40.694470 6149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1214382d30ae4af898176d59577be152. Latest consensus state: current_term: 1 leader_uuid: "1214382d30ae4af898176d59577be152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } } }
I20260807 11:16:40.694674 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1214382d30ae4af898176d59577be152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1214382d30ae4af898176d59577be152" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33825 } } }
I20260807 11:16:40.694953 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:40.694954 6149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:40.706939 6160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:40.711506 6160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:40.719161 6160 catalog_manager.cc:1347] Generated new cluster ID: b6b2fa01f1da433396915476e487c332
I20260807 11:16:40.719269 6160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:40.730050 6160 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:40.731467 6160 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:40.738613 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:40.729058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.2.223.190:33825)
0807 11:16:40.729282 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:40.729291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:40.729896 (+ 605us) server_negotiation.cc:407] Connection header received
0807 11:16:40.730194 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:40.730199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:40.730284 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:40.730403 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:40.731780 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:40.732857 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:40.734755 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:40.735421 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:40.735535 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:40.736759 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:40.736765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:40.736770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:40.737712 (+ 942us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:40.737791 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:40.737823 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:40.737921 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:40.738097 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 11:16:40.738230 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:16:40.742705 6160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1214382d30ae4af898176d59577be152: Generated new TSK 0
I20260807 11:16:40.743467 6160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:16:40.753904 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6072
I20260807 11:16:40.755733 6137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:40.846210 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6072
2026-08-07T11:16:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1977 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6178](info): set up 2 sockets
Aug 07 11:16:40 dist-test-slave-5l3k krb5kdc[6178](info): commencing operation
krb5kdc: starting...
W20260807 11:16:41.980620 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:42 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:42.201750 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:44161
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:42.532653 6193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:42.532987 6193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:42.533143 6193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:42.540454 6193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:42.540596 6193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:42.540699 6193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:42.540798 6193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:42.540870 6193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:42.550731 6193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:44161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:42.552044 6193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:42.553328 6193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:42.564888 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:42.564951 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:42.565136 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:42.566152 6193 server_base.cc:1034] running on GCE node
I20260807 11:16:42.566984 6193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:42.568377 6193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:42.569610 6193 hybrid_clock.cc:648] HybridClock initialized: now 1786101402569571 us; error 42 us; skew 500 ppm
I20260807 11:16:42.569949 6193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:42 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101402, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:42.573033 6193 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:42.574460 6193 webserver.cc:460] Webserver started at http://127.2.223.190:37489/ using document root <none> and password file <none>
I20260807 11:16:42.575137 6193 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:42.575242 6193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:42.575484 6193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:42.578344 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "0a747d0ab2ec4ac5bd631dc260d439e2"
format_stamp: "Formatted at 2026-08-07 11:16:42 on dist-test-slave-5l3k"
server_key: "ab9beb78f9313b9d155b8cbd3d442429"
server_key_iv: "40e69e131ac4edaba653eea5a0f11698"
server_key_version: "encryptionkey@0"
I20260807 11:16:42.579125 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "0a747d0ab2ec4ac5bd631dc260d439e2"
format_stamp: "Formatted at 2026-08-07 11:16:42 on dist-test-slave-5l3k"
server_key: "ab9beb78f9313b9d155b8cbd3d442429"
server_key_iv: "40e69e131ac4edaba653eea5a0f11698"
server_key_version: "encryptionkey@0"
I20260807 11:16:42.584007 6193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:16:42.587388 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:42.588717 6193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:16:42.588900 6193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "0a747d0ab2ec4ac5bd631dc260d439e2"
format_stamp: "Formatted at 2026-08-07 11:16:42 on dist-test-slave-5l3k"
server_key: "ab9beb78f9313b9d155b8cbd3d442429"
server_key_iv: "40e69e131ac4edaba653eea5a0f11698"
server_key_version: "encryptionkey@0"
I20260807 11:16:42.589097 6193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:42.617206 6193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:42.621167 6193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:42.621454 6193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:42.640237 6193 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:44161
I20260807 11:16:42.640273 6261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:44161 every 8 connection(s)
I20260807 11:16:42.641542 6193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:42.645200 6262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:42.651873 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6193
I20260807 11:16:42.652117 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:42.652935 2942 external_mini_cluster.cc:1428] Setting key 81b1c152d31b11b73f71a697176e0e03
I20260807 11:16:42.656484 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2: Bootstrap starting.
I20260807 11:16:42.660005 6262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:42.661347 6262 log.cc:824] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:42.664701 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2: No bootstrap required, opened a new log
Aug 07 11:16:42 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:42.674540 6262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } }
I20260807 11:16:42.675004 6262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:42.675122 6262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a747d0ab2ec4ac5bd631dc260d439e2, State: Initialized, Role: FOLLOWER
I20260807 11:16:42.675837 6262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } }
I20260807 11:16:42.676136 6262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:42.676294 6262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:42.676476 6262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:42.677964 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:42.655079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.2.223.190:44161)
0807 11:16:42.655838 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:42.655850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:42.655891 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:16:42.657647 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:42.657669 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:42.658079 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:42.658398 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:42.660309 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:42.661737 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:42.663649 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:42.664665 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:42.664857 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:42.669008 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:42.669019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:42.669040 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:42.669097 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:42.673224 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:42.673935 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:42.673943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:42.673951 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:42.674168 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:42.674587 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:42.674592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:42.674595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:42.675290 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:42.675686 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:42.676059 (+ 373us) server_negotiation.cc:299] Negotiation successful
0807 11:16:42.676538 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":369,"threads_started":1}
I20260807 11:16:42.679796 6262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } }
I20260807 11:16:42.680279 6262 leader_election.cc:302] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a747d0ab2ec4ac5bd631dc260d439e2; no voters:
I20260807 11:16:42.680930 6262 leader_election.cc:288] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:42.681042 6267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:42.681896 6267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a747d0ab2ec4ac5bd631dc260d439e2, State: Running, Role: LEADER
I20260807 11:16:42.682535 6267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } }
I20260807 11:16:42.683118 6262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:42.685308 6269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a747d0ab2ec4ac5bd631dc260d439e2. Latest consensus state: current_term: 1 leader_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } } }
I20260807 11:16:42.685667 6269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:42.686760 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a747d0ab2ec4ac5bd631dc260d439e2" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44161 } } }
I20260807 11:16:42.687209 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:42.687356 6273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:42.693571 6273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:42.702832 6273 catalog_manager.cc:1347] Generated new cluster ID: 28cfb148068d46dba265ea5cdc39e3be
I20260807 11:16:42.702956 6273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:42.711216 6273 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:42.712448 6273 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:42.730525 6273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a747d0ab2ec4ac5bd631dc260d439e2: Generated new TSK 0
I20260807 11:16:42.731300 6273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:42.785766 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:43.129444 6289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:43.129803 6289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:43.129974 6289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:43.137159 6289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:43.137302 6289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:43.137476 6289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:43.146955 6289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:43.148236 6289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:43.149614 6289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:43.162554 6297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:43.162609 6295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:43.163049 6294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:43.167933 6289 server_base.cc:1034] running on GCE node
I20260807 11:16:43.168483 6289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:43.169405 6289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:43.170627 6289 hybrid_clock.cc:648] HybridClock initialized: now 1786101403170564 us; error 55 us; skew 500 ppm
I20260807 11:16:43.170941 6289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:43 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:43.174705 6289 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:43.176256 6289 webserver.cc:460] Webserver started at http://127.2.223.129:40179/ using document root <none> and password file <none>
I20260807 11:16:43.176960 6289 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:43.177067 6289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:43.177338 6289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:43.180543 6289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "9430f1c1448c4072b5486619f08768c0"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "dd1947868228a9b2ecc1a9049451cd22"
server_key_iv: "81fb7cb504b4c5b6076638feb43e6394"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.181329 6289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "9430f1c1448c4072b5486619f08768c0"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "dd1947868228a9b2ecc1a9049451cd22"
server_key_iv: "81fb7cb504b4c5b6076638feb43e6394"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.186815 6289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:16:43.190146 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.191555 6289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:43.191726 6289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "9430f1c1448c4072b5486619f08768c0"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "dd1947868228a9b2ecc1a9049451cd22"
server_key_iv: "81fb7cb504b4c5b6076638feb43e6394"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.191936 6289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:43.228286 6289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:43.232952 6289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:43.233263 6289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:43.234242 6289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:43.235664 6289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:43.235738 6289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.235904 6289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:43.235963 6289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.270257 6289 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:43167
I20260807 11:16:43.270308 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:43167 every 8 connection(s)
I20260807 11:16:43.271581 6289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:43.278046 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6289
I20260807 11:16:43.278275 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:43.278985 2942 external_mini_cluster.cc:1428] Setting key f7336daca8028398c6eb832ebe7be708
Aug 07 11:16:43 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:43.293350 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:43.274188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:52817 (local address 127.2.223.190:44161)
0807 11:16:43.274674 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:43.274679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:43.275632 (+ 953us) server_negotiation.cc:407] Connection header received
0807 11:16:43.277496 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:43.277501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:43.277588 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:43.277716 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:43.279388 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.280243 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.281911 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.282731 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.282885 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:43.286443 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:43.286452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:43.286459 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:43.286552 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:43.289950 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:43.290923 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:43.290928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:43.290930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:43.291050 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:43.291547 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:43.291550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:43.291552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:43.291840 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:43.291954 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:43.292926 (+ 972us) server_negotiation.cc:299] Negotiation successful
0807 11:16:43.293097 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":223,"threads_started":1}
I20260807 11:16:43.295078 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:43.274773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:44161 (local address 127.2.223.129:52817)
0807 11:16:43.275470 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:43.275518 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:43.277225 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:43.277964 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:43.277973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:43.278622 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:43.279218 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:43.279237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.280410 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.280414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:43.281754 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:43.281767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.282948 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.282953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:43.283227 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:43.283945 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:43.283972 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:43.286214 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:43.290092 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:43.290105 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:43.290122 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:43.290813 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:43.291177 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:43.291184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:43.291190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:43.291434 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:43.291959 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:43.291970 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:43.292406 (+ 436us) client_negotiation.cc:769] Sending connection context
0807 11:16:43.292774 (+ 368us) client_negotiation.cc:240] Negotiation successful
0807 11:16:43.293341 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260807 11:16:43.296777 6423 heartbeater.cc:347] Connected to a master server at 127.2.223.190:44161
I20260807 11:16:43.297103 6423 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:43.297858 6423 heartbeater.cc:511] Master 127.2.223.190:44161 requested a full tablet report, sending...
I20260807 11:16:43.300179 6227 ts_manager.cc:194] Registered new tserver with Master: 9430f1c1448c4072b5486619f08768c0 (127.2.223.129:43167)
I20260807 11:16:43.301667 6227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:52817
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:43.333451 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:43.673578 6429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:43.673858 6429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:43.674017 6429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:43.681751 6429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:43.681886 6429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:43.682108 6429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:43.693109 6429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:43.694574 6429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:43.696023 6429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:43.709316 6436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:43.709277 6435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:43.709337 6438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:43.710925 6429 server_base.cc:1034] running on GCE node
I20260807 11:16:43.711642 6429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:43.712604 6429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:43.713835 6429 hybrid_clock.cc:648] HybridClock initialized: now 1786101403713784 us; error 44 us; skew 500 ppm
I20260807 11:16:43.714165 6429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:43 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:43.717409 6429 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:43.718926 6429 webserver.cc:460] Webserver started at http://127.2.223.130:43167/ using document root <none> and password file <none>
I20260807 11:16:43.719672 6429 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:43.719818 6429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:43.720139 6429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:43.723166 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "3b43ed8669cb4950a82c3f76306d7832"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "c7b58a9dbc278eeb8607b37625d2d23a"
server_key_iv: "aedb5f421fc2843bdbd7cb7236901b36"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.723956 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "3b43ed8669cb4950a82c3f76306d7832"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "c7b58a9dbc278eeb8607b37625d2d23a"
server_key_iv: "aedb5f421fc2843bdbd7cb7236901b36"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.729416 6429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:16:43.732928 6445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.734239 6429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 11:16:43.734413 6429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "3b43ed8669cb4950a82c3f76306d7832"
format_stamp: "Formatted at 2026-08-07 11:16:43 on dist-test-slave-5l3k"
server_key: "c7b58a9dbc278eeb8607b37625d2d23a"
server_key_iv: "aedb5f421fc2843bdbd7cb7236901b36"
server_key_version: "encryptionkey@0"
I20260807 11:16:43.734673 6429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:43.748545 6429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:43.752616 6429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:43.752931 6429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:43.753890 6429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:43.755349 6429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:43.755452 6429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.755599 6429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:43.755688 6429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:43.790092 6429 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:44943
I20260807 11:16:43.790180 6562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:44943 every 8 connection(s)
I20260807 11:16:43.791548 6429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:43.795338 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6429
I20260807 11:16:43.795557 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:43.796139 2942 external_mini_cluster.cc:1428] Setting key ed9fa0b7960da4c1ac2d995c0ff8f810
Aug 07 11:16:43 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:43.816555 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:43.794253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:38595 (local address 127.2.223.190:44161)
0807 11:16:43.794714 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:43.794718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:43.798431 (+ 3713us) server_negotiation.cc:407] Connection header received
0807 11:16:43.798543 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:43.798548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:43.798622 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:43.798714 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:43.800982 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.801717 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.804287 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.804707 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.804810 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:43.810175 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:43.810181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:43.810185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:43.810220 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:43.812978 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:43.814015 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:43.814019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:43.814022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:43.814174 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:43.814713 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:43.814717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:43.814719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:43.814980 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:43.815117 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:43.816094 (+ 977us) server_negotiation.cc:299] Negotiation successful
0807 11:16:43.816272 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260807 11:16:43.818279 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:43.794886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:44161 (local address 127.2.223.130:38595)
0807 11:16:43.795805 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:43.795866 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:43.798028 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:43.799031 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:43.799047 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:43.799907 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:43.800786 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:43.800814 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.801884 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.801892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:43.804111 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:43.804129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:43.804949 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:43.804958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:43.805327 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:43.806146 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:43.806189 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:43.809896 (+ 3707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:43.813133 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:43.813146 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:43.813166 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:43.813884 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:43.814286 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:43.814294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:43.814300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:43.814599 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:43.815133 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:43.815146 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:43.815639 (+ 493us) client_negotiation.cc:769] Sending connection context
0807 11:16:43.815982 (+ 343us) client_negotiation.cc:240] Negotiation successful
0807 11:16:43.816650 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":319,"threads_started":1}
I20260807 11:16:43.819885 6563 heartbeater.cc:347] Connected to a master server at 127.2.223.190:44161
I20260807 11:16:43.820257 6563 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:43.821270 6563 heartbeater.cc:511] Master 127.2.223.190:44161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:16:43.823046 6227 ts_manager.cc:194] Registered new tserver with Master: 3b43ed8669cb4950a82c3f76306d7832 (127.2.223.130:44943)
I20260807 11:16:43.823905 6227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:38595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:43.853806 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:44.183414 6570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:44.183708 6570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:44.183890 6570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:44.191182 6570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:44.191293 6570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:44.191478 6570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:44.201339 6570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:44.202731 6570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:44.204141 6570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:44.217555 6578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:44.217633 6576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:44.217608 6575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:44.218382 6570 server_base.cc:1034] running on GCE node
I20260807 11:16:44.219062 6570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:44.220105 6570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:44.221308 6570 hybrid_clock.cc:648] HybridClock initialized: now 1786101404221243 us; error 52 us; skew 500 ppm
I20260807 11:16:44.221644 6570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:44.224952 6570 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:44.226646 6570 webserver.cc:460] Webserver started at http://127.2.223.131:40615/ using document root <none> and password file <none>
I20260807 11:16:44.227310 6570 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:44.227442 6570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:44.227761 6570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:44.230786 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "57dd713f67e346baae00b71181b2fc3d"
format_stamp: "Formatted at 2026-08-07 11:16:44 on dist-test-slave-5l3k"
server_key: "97d8f13f5fce766adef05c31dd47cee1"
server_key_iv: "ad1334056e03bade756930b6c3efa31c"
server_key_version: "encryptionkey@0"
I20260807 11:16:44.231564 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "57dd713f67e346baae00b71181b2fc3d"
format_stamp: "Formatted at 2026-08-07 11:16:44 on dist-test-slave-5l3k"
server_key: "97d8f13f5fce766adef05c31dd47cee1"
server_key_iv: "ad1334056e03bade756930b6c3efa31c"
server_key_version: "encryptionkey@0"
I20260807 11:16:44.236649 6570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:44.240229 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:44.241531 6570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:16:44.241716 6570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "57dd713f67e346baae00b71181b2fc3d"
format_stamp: "Formatted at 2026-08-07 11:16:44 on dist-test-slave-5l3k"
server_key: "97d8f13f5fce766adef05c31dd47cee1"
server_key_iv: "ad1334056e03bade756930b6c3efa31c"
server_key_version: "encryptionkey@0"
I20260807 11:16:44.241911 6570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:44.255493 6570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:44.259445 6570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:44.259730 6570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:44.260670 6570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:44.261966 6570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:44.262042 6570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:44.262199 6570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:44.262255 6570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260807 11:16:44.274955 6419 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.2.223.129:43167
I20260807 11:16:44.299233 6570 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:46697
I20260807 11:16:44.299293 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:46697 every 8 connection(s)
I20260807 11:16:44.300549 6570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:44.305084 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6570
I20260807 11:16:44.305179 6423 heartbeater.cc:503] Master 127.2.223.190:44161 was elected leader, sending a full tablet report...
I20260807 11:16:44.305279 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:44.305858 2942 external_mini_cluster.cc:1428] Setting key bdf2db1575e45c40f4da761bf76de4cb
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:44.320822 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.303185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:59385 (local address 127.2.223.190:44161)
0807 11:16:44.303404 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:44.303412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:44.304519 (+ 1107us) server_negotiation.cc:407] Connection header received
0807 11:16:44.306754 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:44.306760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:44.306857 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:44.306981 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:44.308630 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.309339 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.310953 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.311337 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.311469 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.314825 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:44.314829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:44.314833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:44.314865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:44.317493 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.318210 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.318219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.318222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.318336 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.318805 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.318810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.318813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.319264 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:44.319448 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:44.320251 (+ 803us) server_negotiation.cc:299] Negotiation successful
0807 11:16:44.320482 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:16:44.321805 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.303651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:44161 (local address 127.2.223.131:59385)
0807 11:16:44.304360 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:44.304397 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:44.306361 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:44.307251 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:44.307261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:44.307920 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:44.308469 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:44.308488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.309485 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.309490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:44.310814 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:44.310826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.311564 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.311571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:44.311792 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.312400 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:44.312422 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:44.314628 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:44.317649 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:44.317657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:44.317672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:44.318084 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:44.318446 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:44.318450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:44.318454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:44.318689 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:44.319415 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:44.319422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:44.319869 (+ 447us) client_negotiation.cc:769] Sending connection context
0807 11:16:44.320134 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 11:16:44.320650 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":293,"threads_started":1}
I20260807 11:16:44.323140 6711 heartbeater.cc:347] Connected to a master server at 127.2.223.190:44161
I20260807 11:16:44.323438 6711 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:44.324159 6711 heartbeater.cc:511] Master 127.2.223.190:44161 requested a full tablet report, sending...
I20260807 11:16:44.325742 6227 ts_manager.cc:194] Registered new tserver with Master: 57dd713f67e346baae00b71181b2fc3d (127.2.223.131:46697)
I20260807 11:16:44.326647 6227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:59385
I20260807 11:16:44.333199 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:44.390314 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.376077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.2.223.190:44161)
0807 11:16:44.376275 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:44.376280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:44.376836 (+ 556us) server_negotiation.cc:407] Connection header received
0807 11:16:44.377055 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:44.377059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:44.377135 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:44.377241 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:44.378635 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.379384 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.380678 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.381096 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.381238 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.385195 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:44.385200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:44.385204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:44.385238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:44.387715 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.388343 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.388350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.388354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.388477 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.388858 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.388864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.388867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.389202 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:44.389466 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:44.389761 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 11:16:44.389979 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 11:16:44.390949 6227 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56060
I20260807 11:16:44.392627 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6289
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:44.408442 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.393677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.2.223.129:43167)
0807 11:16:44.394388 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:44.394396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:44.394420 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:16:44.394521 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:44.394532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:44.394789 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:44.394910 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:44.396200 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.397239 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.398382 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.398991 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.399094 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.403190 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:44.403203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:44.403217 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:44.403250 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:44.405918 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.406666 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.406673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.406675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.406796 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.407193 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.407197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.407198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.407518 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:44.407761 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:44.407956 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 11:16:44.408165 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
W20260807 11:16:44.409668 6418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57634
W20260807 11:16:44.410724 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:44.410950 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6289
I20260807 11:16:44.431159 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6429
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:44.447427 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.432123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.2.223.130:44943)
0807 11:16:44.432668 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:44.432675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:44.432701 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:44.432760 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:44.432767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:44.432966 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:44.433106 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:44.434282 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.435400 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.436640 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.437232 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.437411 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.441597 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:44.441609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:44.441623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:44.441664 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:44.444569 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.445191 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.445198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.445201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.445402 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.445704 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.445708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.445709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.446118 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:44.446341 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:44.446779 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 11:16:44.446998 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":276,"threads_started":1}
W20260807 11:16:44.448844 6556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46926
W20260807 11:16:44.449961 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6429: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:44.450145 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6429
I20260807 11:16:44.470587 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6570
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:44.488231 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:44.471583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.2.223.131:46697)
0807 11:16:44.472226 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:44.472233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:44.472264 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:16:44.472327 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:44.472334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:44.472551 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:44.472732 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:44.474132 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.475365 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.476625 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:44.477188 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:44.477325 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:44.482031 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:44.482045 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:44.482061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:44.482102 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:44.485144 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.485857 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.485865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.485869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.486003 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:44.486414 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:44.486421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:44.486424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:44.486843 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:44.487189 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:44.487525 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 11:16:44.487826 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
W20260807 11:16:44.489584 6705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34844
W20260807 11:16:44.490731 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6570: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:16:44.490911 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6570
I20260807 11:16:44.510759 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6193
I20260807 11:16:44.511925 6257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:44.617888 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6193
2026-08-07T11:16:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3778 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6730](info): set up 2 sockets
Aug 07 11:16:44 dist-test-slave-5l3k krb5kdc[6730](info): commencing operation
krb5kdc: starting...
W20260807 11:16:45.751391 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:45 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:45.979105 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42027
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:42027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:46.305339 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:46.305611 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:46.305773 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:46.312918 6744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:46.313035 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:46.313143 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:46.313241 6744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:46.313318 6744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:46.322974 6744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:42027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:46.324239 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:46.325539 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:46.338690 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:46.338724 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:46.338536 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:46.339680 6744 server_base.cc:1034] running on GCE node
I20260807 11:16:46.340483 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:46.341864 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:46.343096 6744 hybrid_clock.cc:648] HybridClock initialized: now 1786101406343049 us; error 68 us; skew 500 ppm
I20260807 11:16:46.343452 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:46 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:46.350221 6744 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:46.351788 6744 webserver.cc:460] Webserver started at http://127.2.223.190:35397/ using document root <none> and password file <none>
I20260807 11:16:46.352478 6744 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:46.352632 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:46.352945 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:46.355949 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "9281a616b33d4a7f8185467eb729e4e7"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "7908ac7e153ee282bfa9c047c81a1b47"
server_key_iv: "0d818a6ad0fa11b5409a4253ff74537b"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.356742 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "9281a616b33d4a7f8185467eb729e4e7"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "7908ac7e153ee282bfa9c047c81a1b47"
server_key_iv: "0d818a6ad0fa11b5409a4253ff74537b"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.361822 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:16:46.365185 6760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:46.366397 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:46.366628 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "9281a616b33d4a7f8185467eb729e4e7"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "7908ac7e153ee282bfa9c047c81a1b47"
server_key_iv: "0d818a6ad0fa11b5409a4253ff74537b"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.366832 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:46.380281 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:46.384192 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:46.384465 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:46.403260 6744 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:42027
I20260807 11:16:46.403307 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:42027 every 8 connection(s)
I20260807 11:16:46.404608 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:46.408255 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:46.410065 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6744
I20260807 11:16:46.410333 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:46.411046 2942 external_mini_cluster.cc:1428] Setting key 532286543f14c8a89583ea6de230316d
I20260807 11:16:46.420228 6815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7: Bootstrap starting.
I20260807 11:16:46.424041 6815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:46.425364 6815 log.cc:824] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:16:46 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:46.428131 6815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7: No bootstrap required, opened a new log
I20260807 11:16:46.434437 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:46.413038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.2.223.190:42027)
0807 11:16:46.413950 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:46.413961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:46.414000 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:16:46.415700 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:46.415723 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:46.416109 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:46.416426 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:46.418317 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:46.419676 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:46.421419 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:46.422086 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:46.422221 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:46.427184 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:46.427192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:46.427206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:46.427238 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:46.430460 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:46.431157 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:46.431165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:46.431177 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:46.431355 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:46.431740 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:46.431746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:46.431749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:46.432256 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:46.432593 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:46.433064 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 11:16:46.433475 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":402,"threads_started":1}
I20260807 11:16:46.435971 6815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } }
I20260807 11:16:46.436365 6815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:46.436489 6815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9281a616b33d4a7f8185467eb729e4e7, State: Initialized, Role: FOLLOWER
I20260807 11:16:46.437122 6815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } }
I20260807 11:16:46.437405 6815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:46.437533 6815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:46.437656 6815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:46.439831 6815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } }
I20260807 11:16:46.440407 6815 leader_election.cc:302] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9281a616b33d4a7f8185467eb729e4e7; no voters:
I20260807 11:16:46.440986 6815 leader_election.cc:288] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:46.441133 6820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:46.441890 6820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9281a616b33d4a7f8185467eb729e4e7, State: Running, Role: LEADER
I20260807 11:16:46.442381 6820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } }
I20260807 11:16:46.443174 6815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:46.445319 6822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9281a616b33d4a7f8185467eb729e4e7. Latest consensus state: current_term: 1 leader_uuid: "9281a616b33d4a7f8185467eb729e4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } } }
I20260807 11:16:46.445787 6822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:46.446430 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9281a616b33d4a7f8185467eb729e4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9281a616b33d4a7f8185467eb729e4e7" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42027 } } }
I20260807 11:16:46.446807 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:46.451454 6826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:46.456593 6826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:46.465574 6826 catalog_manager.cc:1347] Generated new cluster ID: b6617a159f1242b5976688a7528ffd9d
I20260807 11:16:46.465677 6826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:46.487517 6826 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:46.488785 6826 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:46.504094 6826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9281a616b33d4a7f8185467eb729e4e7: Generated new TSK 0
I20260807 11:16:46.504981 6826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:46.565272 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:46.920042 6843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:46.920387 6843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:46.920565 6843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:46.928810 6843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:46.928973 6843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:46.929177 6843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:46.940155 6843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:46.941557 6843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:46.943094 6843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:46.957513 6848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:46.957676 6851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:46.957697 6849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:46.958386 6843 server_base.cc:1034] running on GCE node
I20260807 11:16:46.959163 6843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:46.960224 6843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:46.961464 6843 hybrid_clock.cc:648] HybridClock initialized: now 1786101406961424 us; error 57 us; skew 500 ppm
I20260807 11:16:46.961833 6843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:46 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:46.965211 6843 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:46.966825 6843 webserver.cc:460] Webserver started at http://127.2.223.129:44667/ using document root <none> and password file <none>
I20260807 11:16:46.967571 6843 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:46.967726 6843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:46.968115 6843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:46.971660 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "2ae75bac6c514624a2b9c520a33decdf"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "8f1327185a344e1344a52d9c63ed99e8"
server_key_iv: "46ed0c15bb9b6e29486501d82cb878cb"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.972496 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "2ae75bac6c514624a2b9c520a33decdf"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "8f1327185a344e1344a52d9c63ed99e8"
server_key_iv: "46ed0c15bb9b6e29486501d82cb878cb"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.977764 6843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:16:46.981271 6858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:46.982612 6843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:46.982776 6843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "2ae75bac6c514624a2b9c520a33decdf"
format_stamp: "Formatted at 2026-08-07 11:16:46 on dist-test-slave-5l3k"
server_key: "8f1327185a344e1344a52d9c63ed99e8"
server_key_iv: "46ed0c15bb9b6e29486501d82cb878cb"
server_key_version: "encryptionkey@0"
I20260807 11:16:46.982980 6843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:47.017537 6843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:47.021670 6843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:47.021983 6843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:47.023020 6843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:47.024379 6843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:47.024482 6843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:47.024647 6843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:47.024722 6843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:47.059043 6843 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:34115
I20260807 11:16:47.059131 6976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:34115 every 8 connection(s)
I20260807 11:16:47.060400 6843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:47.069988 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6843
I20260807 11:16:47.070220 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:47.071041 2942 external_mini_cluster.cc:1428] Setting key a5390d32701e64396e8f07b649c7b3c2
Aug 07 11:16:47 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:47.081373 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:47.062963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:51595 (local address 127.2.223.190:42027)
0807 11:16:47.063433 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:47.063437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:47.064302 (+ 865us) server_negotiation.cc:407] Connection header received
0807 11:16:47.066238 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:47.066244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:47.066335 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:47.066450 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:47.067940 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.068674 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.070793 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.071276 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.071380 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:47.075657 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:47.075663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:47.075666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:47.075697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:47.078284 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:47.079057 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:47.079069 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:47.079071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:47.079199 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:47.079651 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:47.079655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:47.079657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:47.079905 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:47.080093 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:47.080753 (+ 660us) server_negotiation.cc:299] Negotiation successful
0807 11:16:47.080973 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260807 11:16:47.082631 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:47.063433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42027 (local address 127.2.223.129:51595)
0807 11:16:47.064140 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:47.064189 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:47.065961 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:47.066712 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:47.066720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:47.067210 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:47.067775 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:47.067792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.068850 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.068854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:47.070638 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:47.070655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.071525 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.071534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:47.071800 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:47.072535 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:47.072571 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:47.075430 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:47.078429 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:47.078438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:47.078454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:47.078928 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:47.079322 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:47.079327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:47.079330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:47.079520 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:47.080037 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:47.080045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:47.080386 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 11:16:47.080665 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 11:16:47.081193 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":288,"threads_started":1}
I20260807 11:16:47.084180 6977 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42027
I20260807 11:16:47.084484 6977 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:47.085197 6977 heartbeater.cc:511] Master 127.2.223.190:42027 requested a full tablet report, sending...
I20260807 11:16:47.087613 6777 ts_manager.cc:194] Registered new tserver with Master: 2ae75bac6c514624a2b9c520a33decdf (127.2.223.129:34115)
I20260807 11:16:47.089823 6777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:51595
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:47.125895 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:47.456365 6984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:47.456669 6984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:47.456761 6984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:47.463737 6984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:47.463856 6984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:47.463966 6984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:47.473590 6984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:47.474824 6984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:47.476122 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:47.489389 6989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:47.489343 6992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:47.489297 6990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:47.490160 6984 server_base.cc:1034] running on GCE node
I20260807 11:16:47.490841 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:47.491776 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:47.493019 6984 hybrid_clock.cc:648] HybridClock initialized: now 1786101407492974 us; error 51 us; skew 500 ppm
I20260807 11:16:47.493326 6984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:47 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101407, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:47.496577 6984 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:47.498056 6984 webserver.cc:460] Webserver started at http://127.2.223.130:33673/ using document root <none> and password file <none>
I20260807 11:16:47.498772 6984 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:47.498915 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:47.499226 6984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:47.502090 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "69168ed5e04e4f478408cec4b7a76a44"
format_stamp: "Formatted at 2026-08-07 11:16:47 on dist-test-slave-5l3k"
server_key: "f4d73c7e4de2af84b47b03e1861f8f99"
server_key_iv: "c082e6c9ff7fbd848bd6ac1f4d9a795b"
server_key_version: "encryptionkey@0"
I20260807 11:16:47.502880 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "69168ed5e04e4f478408cec4b7a76a44"
format_stamp: "Formatted at 2026-08-07 11:16:47 on dist-test-slave-5l3k"
server_key: "f4d73c7e4de2af84b47b03e1861f8f99"
server_key_iv: "c082e6c9ff7fbd848bd6ac1f4d9a795b"
server_key_version: "encryptionkey@0"
I20260807 11:16:47.508152 6984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:16:47.511669 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:47.512965 6984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:16:47.513188 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "69168ed5e04e4f478408cec4b7a76a44"
format_stamp: "Formatted at 2026-08-07 11:16:47 on dist-test-slave-5l3k"
server_key: "f4d73c7e4de2af84b47b03e1861f8f99"
server_key_iv: "c082e6c9ff7fbd848bd6ac1f4d9a795b"
server_key_version: "encryptionkey@0"
I20260807 11:16:47.513409 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:47.541455 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:47.545951 6984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:47.546259 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:47.547354 6984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:47.548681 6984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:47.548785 6984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:47.548934 6984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:47.549006 6984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:47.583290 6984 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:45423
I20260807 11:16:47.583344 7118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:45423 every 8 connection(s)
I20260807 11:16:47.584677 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:47.587561 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 6984
I20260807 11:16:47.587791 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:47.588224 2942 external_mini_cluster.cc:1428] Setting key defd165467c885ae9e5129cbac35a5b3
Aug 07 11:16:47 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101407, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:47.609835 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:47.587447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:60875 (local address 127.2.223.190:42027)
0807 11:16:47.587952 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:47.587958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:47.589398 (+ 1440us) server_negotiation.cc:407] Connection header received
0807 11:16:47.592006 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:47.592010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:47.592080 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:47.592171 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:47.594304 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.595101 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.597584 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.598072 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.598177 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:47.602599 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:47.602608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:47.602615 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:47.602660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:47.606423 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:47.607209 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:47.607217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:47.607223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:47.607385 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:47.607805 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:47.607812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:47.607818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:47.608208 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:47.608382 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:47.609042 (+ 660us) server_negotiation.cc:299] Negotiation successful
0807 11:16:47.609319 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260807 11:16:47.610518 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:47.588078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42027 (local address 127.2.223.130:60875)
0807 11:16:47.589227 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:47.589288 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:47.591713 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:47.592447 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:47.592460 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:47.593295 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:47.594125 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:47.594154 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.595254 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.595262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:47.597421 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:47.597445 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:47.598388 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:47.598400 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:47.598801 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:47.599602 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:47.599640 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:47.602333 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:47.606611 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:47.606618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:47.606631 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:47.607049 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:47.607503 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:47.607506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:47.607509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:47.607689 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:47.608337 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:47.608343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:47.608639 (+ 296us) client_negotiation.cc:769] Sending connection context
0807 11:16:47.608881 (+ 242us) client_negotiation.cc:240] Negotiation successful
0807 11:16:47.609271 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":665,"threads_started":1}
I20260807 11:16:47.612026 7119 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42027
I20260807 11:16:47.612351 7119 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:47.613058 7119 heartbeater.cc:511] Master 127.2.223.190:42027 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:16:47.614614 6777 ts_manager.cc:194] Registered new tserver with Master: 69168ed5e04e4f478408cec4b7a76a44 (127.2.223.130:45423)
I20260807 11:16:47.615497 6777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:60875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:47.644932 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:47.976902 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:47.977200 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:47.977366 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:47.984753 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:47.984879 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:47.985064 7125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:47.994902 7125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:47.996150 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:47.997491 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:48.010941 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:48.011235 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:48.011461 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:48.011937 7125 server_base.cc:1034] running on GCE node
I20260807 11:16:48.012610 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:48.013499 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:48.014708 7125 hybrid_clock.cc:648] HybridClock initialized: now 1786101408014647 us; error 42 us; skew 500 ppm
I20260807 11:16:48.015025 7125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:48.018445 7125 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:48.019905 7125 webserver.cc:460] Webserver started at http://127.2.223.131:39321/ using document root <none> and password file <none>
I20260807 11:16:48.020534 7125 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:48.020640 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:48.020893 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:48.023774 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "23ed8dbae9b24eee9cdc0234e9f0115a"
format_stamp: "Formatted at 2026-08-07 11:16:48 on dist-test-slave-5l3k"
server_key: "51471c5ca24699cc519a20075d8accac"
server_key_iv: "ae1d6ea271ac077ba7c5b60f765318a0"
server_key_version: "encryptionkey@0"
I20260807 11:16:48.024502 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "23ed8dbae9b24eee9cdc0234e9f0115a"
format_stamp: "Formatted at 2026-08-07 11:16:48 on dist-test-slave-5l3k"
server_key: "51471c5ca24699cc519a20075d8accac"
server_key_iv: "ae1d6ea271ac077ba7c5b60f765318a0"
server_key_version: "encryptionkey@0"
I20260807 11:16:48.029271 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 11:16:48.032615 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:48.033939 7125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 11:16:48.034140 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "23ed8dbae9b24eee9cdc0234e9f0115a"
format_stamp: "Formatted at 2026-08-07 11:16:48 on dist-test-slave-5l3k"
server_key: "51471c5ca24699cc519a20075d8accac"
server_key_iv: "ae1d6ea271ac077ba7c5b60f765318a0"
server_key_version: "encryptionkey@0"
I20260807 11:16:48.034337 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:48.061610 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:48.065635 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:48.065913 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:48.066839 7125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:48.068112 7125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:48.068212 7125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:48.068336 7125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:48.068400 7125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:48.092818 6977 heartbeater.cc:503] Master 127.2.223.190:42027 was elected leader, sending a full tablet report...
I20260807 11:16:48.103202 7125 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:46251
I20260807 11:16:48.103282 7259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:46251 every 8 connection(s)
I20260807 11:16:48.104483 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:48.106456 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7125
I20260807 11:16:48.106704 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:48.107165 2942 external_mini_cluster.cc:1428] Setting key 7b6d3676886cb3e67bb00a2d77a0e686
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:48.124925 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.107312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:35355 (local address 127.2.223.190:42027)
0807 11:16:48.107522 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.107531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.109033 (+ 1502us) server_negotiation.cc:407] Connection header received
0807 11:16:48.111363 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.111367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.111444 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.111528 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.112940 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.113686 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.115225 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.115616 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.115707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.118967 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.118972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.118975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.119006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.122140 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.122980 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.122984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.122987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.123118 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.123504 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.123507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.123509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.123775 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.123884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.124524 (+ 640us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.124694 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:16:48.125857 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.107986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42027 (local address 127.2.223.131:35355)
0807 11:16:48.108816 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:48.108901 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:48.111125 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:48.111739 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:48.111746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:48.112247 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:48.112797 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:48.112814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.113817 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.113821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:48.115105 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:48.115122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.115790 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.115794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:48.115969 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.116564 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:48.116618 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:48.118777 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:48.122272 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:48.122278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:48.122288 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:48.122865 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:48.123216 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:48.123219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:48.123222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:48.123411 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:48.123914 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:48.123920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:48.124206 (+ 286us) client_negotiation.cc:769] Sending connection context
0807 11:16:48.124443 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 11:16:48.124796 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":319,"threads_started":1}
I20260807 11:16:48.127292 7260 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42027
I20260807 11:16:48.127554 7260 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:48.128194 7260 heartbeater.cc:511] Master 127.2.223.190:42027 requested a full tablet report, sending...
I20260807 11:16:48.129585 6777 ts_manager.cc:194] Registered new tserver with Master: 23ed8dbae9b24eee9cdc0234e9f0115a (127.2.223.131:46251)
I20260807 11:16:48.130359 6777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:35355
I20260807 11:16:48.134784 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:48.189520 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.174748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.2.223.129:34115)
0807 11:16:48.175417 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.175424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.175449 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:16:48.175509 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.175515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.175761 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.175887 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.177303 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.178315 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.179634 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.180167 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.180263 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.184182 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.184193 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.184208 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.184243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.187120 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.187780 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.187787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.187789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.187909 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.188251 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.188254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.188256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.188561 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.188826 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.189057 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.189243 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":376,"threads_started":1}
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:48.207424 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.193927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.2.223.190:42027)
0807 11:16:48.194138 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.194144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.194346 (+ 202us) server_negotiation.cc:407] Connection header received
0807 11:16:48.194460 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.194464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.194604 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.194684 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.195860 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.196595 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.197768 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.198165 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.198261 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.201846 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.201851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.201854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.201886 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.204558 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.205283 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.205287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.205290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.205426 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.205976 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.205980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.205982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.206332 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.206624 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.206877 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.207045 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 11:16:48.208019 6777 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48900
I20260807 11:16:48.208981 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6843
I20260807 11:16:48.221678 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.209920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.2.223.129:34115)
0807 11:16:48.210129 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.210135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.210317 (+ 182us) server_negotiation.cc:407] Connection header received
0807 11:16:48.210535 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.210540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.210624 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.210728 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.211933 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.212622 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.213833 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.214279 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.214411 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.216619 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.216624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.216628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.216660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.219184 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.219797 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.219804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.219807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.219938 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.220307 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.220312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.220316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.220569 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.220837 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.221139 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.221356 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:16:48.222287 6972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:48.305039 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6843
I20260807 11:16:48.325627 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6984
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:48.341403 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.326573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.2.223.130:45423)
0807 11:16:48.327112 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.327119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.327147 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:16:48.327210 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.327216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.327448 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.327574 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.328778 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.329811 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.331192 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.331734 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.331832 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.335996 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.336007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.336023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.336062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.339012 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.339619 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.339626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.339628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.339758 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.340075 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.340079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.340080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.340403 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.340641 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.340877 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.341105 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":265,"threads_started":1}
I20260807 11:16:48.342708 7114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:48.427052 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6984
I20260807 11:16:48.448897 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7125
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[6730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:48.466801 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:48.450129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.2.223.131:46251)
0807 11:16:48.450831 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:48.450838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:48.450870 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:16:48.450931 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:48.450938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:48.451181 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:48.451324 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:48.452638 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.453861 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.455336 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:48.455903 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:48.456019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:48.460572 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:48.460587 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:48.460602 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:48.460639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:48.463781 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.464548 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.464556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.464558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.464687 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:48.465069 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:48.465074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:48.465077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:48.465447 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:48.465781 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:48.466106 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 11:16:48.466396 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":379,"threads_started":1}
I20260807 11:16:48.468179 7255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:48.551980 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7125
I20260807 11:16:48.572858 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 6744
I20260807 11:16:48.574128 6809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:48.681399 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 6744
2026-08-07T11:16:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4061 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[7280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[7280](info): set up 2 sockets
Aug 07 11:16:48 dist-test-slave-5l3k krb5kdc[7280](info): commencing operation
krb5kdc: starting...
W20260807 11:16:49.827911 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:16:49 dist-test-slave-5l3k krb5kdc[7280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:16:50.077936 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45261
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:50.404307 7293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:50.404606 7293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:50.404737 7293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:50.411962 7293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:50.412062 7293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:50.412177 7293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:50.412276 7293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:50.412376 7293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:50.422017 7293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:50.423311 7293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:50.424602 7293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:50.436136 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:50.436193 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:50.436170 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:50.437448 7293 server_base.cc:1034] running on GCE node
I20260807 11:16:50.438160 7293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:50.439625 7293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:50.440824 7293 hybrid_clock.cc:648] HybridClock initialized: now 1786101410440767 us; error 61 us; skew 500 ppm
I20260807 11:16:50.441180 7293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:50 dist-test-slave-5l3k krb5kdc[7280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:50.444227 7293 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:16:50.445278 7293 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:50.445375 7293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:50.445626 7293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:50.448496 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "65efb51fab03400d99e1968ac3402f59"
format_stamp: "Formatted at 2026-08-07 11:16:50 on dist-test-slave-5l3k"
server_key: "1ba911bee642730bc3a7de3b79d158bc"
server_key_iv: "ed698a9704e6e1ba5da1b9839e4e5364"
server_key_version: "encryptionkey@0"
I20260807 11:16:50.449241 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "65efb51fab03400d99e1968ac3402f59"
format_stamp: "Formatted at 2026-08-07 11:16:50 on dist-test-slave-5l3k"
server_key: "1ba911bee642730bc3a7de3b79d158bc"
server_key_iv: "ed698a9704e6e1ba5da1b9839e4e5364"
server_key_version: "encryptionkey@0"
I20260807 11:16:50.454037 7293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260807 11:16:50.457374 7307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:50.458709 7293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:50.458843 7293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "65efb51fab03400d99e1968ac3402f59"
format_stamp: "Formatted at 2026-08-07 11:16:50 on dist-test-slave-5l3k"
server_key: "1ba911bee642730bc3a7de3b79d158bc"
server_key_iv: "ed698a9704e6e1ba5da1b9839e4e5364"
server_key_version: "encryptionkey@0"
I20260807 11:16:50.459039 7293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:50.491250 7293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:50.495888 7293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:50.496220 7293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:50.515187 7293 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:45261
I20260807 11:16:50.515208 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:45261 every 8 connection(s)
I20260807 11:16:50.516101 7293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:50.518622 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7293
I20260807 11:16:50.518867 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:50.519533 2942 external_mini_cluster.cc:1428] Setting key 31833b94cc685921e98df41153fb7296
I20260807 11:16:50.519883 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:50.531952 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: Bootstrap starting.
Aug 07 11:16:50 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:50.535055 7361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:50.536309 7361 log.cc:824] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:50.539204 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: No bootstrap required, opened a new log
I20260807 11:16:50.542702 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:50.521569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.2.223.190:45261)
0807 11:16:50.522374 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:50.522389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:50.522428 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:16:50.524546 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:50.524570 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:50.524951 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:50.525352 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:50.526961 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:50.528276 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:50.529636 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:50.530360 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:50.530588 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:50.534580 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:50.534589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:50.534604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:50.534639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:50.537926 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:50.538878 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:50.538885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:50.538891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:50.539086 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:50.539573 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:50.539578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:50.539580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:50.540243 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:50.540574 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:50.541150 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 11:16:50.541566 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":351,"threads_started":1}
I20260807 11:16:50.547204 7361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } }
I20260807 11:16:50.547565 7361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:50.547673 7361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65efb51fab03400d99e1968ac3402f59, State: Initialized, Role: FOLLOWER
I20260807 11:16:50.548262 7361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } }
I20260807 11:16:50.548507 7361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:50.548646 7361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:50.548826 7361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:50.551403 7361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } }
I20260807 11:16:50.551903 7361 leader_election.cc:302] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65efb51fab03400d99e1968ac3402f59; no voters:
I20260807 11:16:50.552472 7361 leader_election.cc:288] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:50.552631 7366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:50.553341 7366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [term 1 LEADER]: Becoming Leader. State: Replica: 65efb51fab03400d99e1968ac3402f59, State: Running, Role: LEADER
I20260807 11:16:50.553925 7366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } }
I20260807 11:16:50.554600 7361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:50.556734 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65efb51fab03400d99e1968ac3402f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } } }
I20260807 11:16:50.557140 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:50.559516 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65efb51fab03400d99e1968ac3402f59. Latest consensus state: current_term: 1 leader_uuid: "65efb51fab03400d99e1968ac3402f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65efb51fab03400d99e1968ac3402f59" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45261 } } }
I20260807 11:16:50.560000 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:50.561231 7373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:50.568090 7373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:50.577471 7373 catalog_manager.cc:1347] Generated new cluster ID: 2136aabbbbae40b3a1d1188c8270aeec
I20260807 11:16:50.577623 7373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:50.592782 7373 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:50.594067 7373 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:50.605558 7373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: Generated new TSK 0
I20260807 11:16:50.606361 7373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:16:50.663136 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:51.006366 7389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:51.006738 7389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:51.006919 7389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:51.014256 7389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:51.014412 7389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:51.014639 7389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:51.024468 7389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:51.025835 7389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:51.027201 7389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:51.039842 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:51.039908 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:51.040522 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:51.041518 7389 server_base.cc:1034] running on GCE node
I20260807 11:16:51.042120 7389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:51.043073 7389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:51.044293 7389 hybrid_clock.cc:648] HybridClock initialized: now 1786101411044255 us; error 35 us; skew 500 ppm
I20260807 11:16:51.044612 7389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:51 dist-test-slave-5l3k krb5kdc[7280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101411, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:51.047986 7389 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:16:51.049226 7389 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:51.049362 7389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:51.049683 7389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:51.052636 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "c7c9bb6a5f9f0b8a56c08b03ecd6ab3c"
server_key_iv: "8cee7c11a41ef08a15d89d837f936f23"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.053470 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "c7c9bb6a5f9f0b8a56c08b03ecd6ab3c"
server_key_iv: "8cee7c11a41ef08a15d89d837f936f23"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.058768 7389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:16:51.062358 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.063762 7389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:51.063937 7389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "c7c9bb6a5f9f0b8a56c08b03ecd6ab3c"
server_key_iv: "8cee7c11a41ef08a15d89d837f936f23"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.064193 7389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:51.075646 7389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:51.079936 7389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:51.080242 7389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:51.081192 7389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:51.082602 7389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:51.082718 7389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.082851 7389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:51.082939 7389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.117937 7389 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:37407
I20260807 11:16:51.118198 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:37407 every 8 connection(s)
I20260807 11:16:51.118950 7389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:51.126816 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7389
I20260807 11:16:51.127036 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:51.127672 2942 external_mini_cluster.cc:1428] Setting key ede3914075b521a07ceaa129c6fc8116
Aug 07 11:16:51 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101411, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:51.138518 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:51.121357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38793 (local address 127.2.223.190:45261)
0807 11:16:51.121795 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:51.121800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:51.122821 (+ 1021us) server_negotiation.cc:407] Connection header received
0807 11:16:51.124708 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:51.124713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:51.124800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:51.124917 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:51.126340 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.127207 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.128927 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.129477 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.129638 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:51.133035 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:51.133040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:51.133044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:51.133076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:51.135598 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:51.136268 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:51.136275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:51.136278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:51.136403 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:51.136809 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:51.136815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:51.136818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:51.137077 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:51.137246 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:51.137911 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 11:16:51.138131 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260807 11:16:51.139411 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:51.121937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45261 (local address 127.2.223.129:38793)
0807 11:16:51.122671 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:51.122710 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:51.124445 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:51.125152 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:51.125161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:51.125645 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:51.126171 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:51.126188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.127368 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.127372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:51.128784 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:51.128796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.129726 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.129736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:51.129964 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:51.130688 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:51.130709 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:51.132820 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:51.135732 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:51.135740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:51.135753 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:51.136153 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:51.136512 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:51.136516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:51.136520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:51.136699 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:51.137222 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:51.137229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:51.137528 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 11:16:51.137806 (+ 278us) client_negotiation.cc:240] Negotiation successful
0807 11:16:51.138287 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260807 11:16:51.141090 7521 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45261
I20260807 11:16:51.141399 7521 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:51.142191 7521 heartbeater.cc:511] Master 127.2.223.190:45261 requested a full tablet report, sending...
I20260807 11:16:51.144693 7325 ts_manager.cc:194] Registered new tserver with Master: 1c94dd7b4cdb4be381d1df261b0fb4e6 (127.2.223.129:37407)
I20260807 11:16:51.146872 7325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:38793
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:16:51.181793 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:51.508584 7527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:51.508878 7527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:51.508980 7527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:51.517297 7527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:51.517427 7527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:51.517613 7527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:51.533107 7527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:51.534668 7527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:51.536243 7527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:51.549795 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:51.549837 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:51.549898 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:51.550913 7527 server_base.cc:1034] running on GCE node
I20260807 11:16:51.551643 7527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:51.552752 7527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:51.553995 7527 hybrid_clock.cc:648] HybridClock initialized: now 1786101411553945 us; error 29 us; skew 500 ppm
I20260807 11:16:51.554308 7527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:51 dist-test-slave-5l3k krb5kdc[7280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101411, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:51.563288 7527 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:16:51.564584 7527 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:51.564730 7527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:51.565078 7527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:51.568660 7527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "37a90a9b8251410ab791cca5a34b7d9d"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "3785680a4f0db69881afff6bb77926d5"
server_key_iv: "63208fa8bc23e6f15d754837440c3b50"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.569656 7527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "37a90a9b8251410ab791cca5a34b7d9d"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "3785680a4f0db69881afff6bb77926d5"
server_key_iv: "63208fa8bc23e6f15d754837440c3b50"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.575135 7527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:16:51.578675 7541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.580125 7527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:16:51.580323 7527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "37a90a9b8251410ab791cca5a34b7d9d"
format_stamp: "Formatted at 2026-08-07 11:16:51 on dist-test-slave-5l3k"
server_key: "3785680a4f0db69881afff6bb77926d5"
server_key_iv: "63208fa8bc23e6f15d754837440c3b50"
server_key_version: "encryptionkey@0"
I20260807 11:16:51.580554 7527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:51.603948 7527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:51.608413 7527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:51.608765 7527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:51.609858 7527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:51.611423 7527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:51.611558 7527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.611706 7527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:51.611804 7527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:51.650156 7527 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:43453
I20260807 11:16:51.650251 7661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:43453 every 8 connection(s)
I20260807 11:16:51.651093 7527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:51.653476 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7527
I20260807 11:16:51.653689 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:51.654316 2942 external_mini_cluster.cc:1428] Setting key 1daf422065279cb2ab85d5419d530cff
Aug 07 11:16:51 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101411, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:51.673758 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:51.653820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:37475 (local address 127.2.223.190:45261)
0807 11:16:51.654354 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:51.654360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:51.655697 (+ 1337us) server_negotiation.cc:407] Connection header received
0807 11:16:51.657741 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:51.657747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:51.657833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:51.657966 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:51.659556 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.660298 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.661979 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.662597 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.662735 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:51.666438 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:51.666446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:51.666452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:51.666525 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:51.670307 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:51.671081 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:51.671089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:51.671096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:51.671268 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:51.671713 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:51.671721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:51.671727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:51.672121 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:51.672276 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:51.672951 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 11:16:51.673245 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260807 11:16:51.674307 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:51.654535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45261 (local address 127.2.223.130:37475)
0807 11:16:51.655522 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:51.655568 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:51.657460 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:51.658209 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:51.658217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:51.658797 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:51.659373 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:51.659391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.660493 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.660500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:51.661837 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:51.661849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:51.662838 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:51.662845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:51.663061 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:51.663786 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:51.663810 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:51.666194 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:51.670453 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:51.670460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:51.670472 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:51.670955 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:51.671381 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:51.671383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:51.671386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:51.671572 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:51.672241 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:51.672246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:51.672567 (+ 321us) client_negotiation.cc:769] Sending connection context
0807 11:16:51.672788 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 11:16:51.673178 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":386,"threads_started":1}
I20260807 11:16:51.675848 7662 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45261
I20260807 11:16:51.676198 7662 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:51.677021 7662 heartbeater.cc:511] Master 127.2.223.190:45261 requested a full tablet report, sending...
I20260807 11:16:51.678834 7325 ts_manager.cc:194] Registered new tserver with Master: 37a90a9b8251410ab791cca5a34b7d9d (127.2.223.130:43453)
I20260807 11:16:51.679715 7325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:37475
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:16:51.713557 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:16:52.088434 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:52.088734 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:52.088905 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:52.096077 7668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:52.096220 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:52.096401 7668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:52.106040 7668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:52.107417 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:52.108781 7668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:52.122300 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:52.122308 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:52.122848 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:52.126295 7668 server_base.cc:1034] running on GCE node
I20260807 11:16:52.126989 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:52.127941 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:52.129246 7668 hybrid_clock.cc:648] HybridClock initialized: now 1786101412129172 us; error 60 us; skew 500 ppm
I20260807 11:16:52.129624 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:16:52 dist-test-slave-5l3k krb5kdc[7280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101412, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:16:52.132812 7668 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:16:52.134042 7668 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:52.134179 7668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:52.134558 7668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:52.137503 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "8c1960357c224fc2a1d850d27e36d031"
format_stamp: "Formatted at 2026-08-07 11:16:52 on dist-test-slave-5l3k"
server_key: "5369dc93db99a633ab29a98bf37fa5a3"
server_key_iv: "c341eb02e040eab0ceeb7285a8194b7b"
server_key_version: "encryptionkey@0"
I20260807 11:16:52.138341 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "8c1960357c224fc2a1d850d27e36d031"
format_stamp: "Formatted at 2026-08-07 11:16:52 on dist-test-slave-5l3k"
server_key: "5369dc93db99a633ab29a98bf37fa5a3"
server_key_iv: "c341eb02e040eab0ceeb7285a8194b7b"
server_key_version: "encryptionkey@0"
I20260807 11:16:52.143546 7668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:16:52.147065 7683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:52.148411 7668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:16:52.148600 7668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "8c1960357c224fc2a1d850d27e36d031"
format_stamp: "Formatted at 2026-08-07 11:16:52 on dist-test-slave-5l3k"
server_key: "5369dc93db99a633ab29a98bf37fa5a3"
server_key_iv: "c341eb02e040eab0ceeb7285a8194b7b"
server_key_version: "encryptionkey@0"
I20260807 11:16:52.148834 7668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:52.150781 7521 heartbeater.cc:503] Master 127.2.223.190:45261 was elected leader, sending a full tablet report...
I20260807 11:16:52.178170 7668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:52.182456 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:52.182823 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:52.184003 7668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:52.185609 7668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:52.185731 7668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:52.185864 7668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:52.185956 7668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:52.221788 7668 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:43395
I20260807 11:16:52.221882 7801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:43395 every 8 connection(s)
I20260807 11:16:52.222790 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:52.229255 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7668
I20260807 11:16:52.229504 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:52.230079 2942 external_mini_cluster.cc:1428] Setting key 7943f6b9f1b38c19810383a1d9558f89
Aug 07 11:16:52 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101412, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:16:52.244383 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.225456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:54871 (local address 127.2.223.190:45261)
0807 11:16:52.225909 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.225913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.227007 (+ 1094us) server_negotiation.cc:407] Connection header received
0807 11:16:52.229055 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.229060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.229152 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.229253 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:52.231325 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.232289 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.234337 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.234808 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.234909 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.238605 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:52.238610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:52.238614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:52.238649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:52.241225 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.242015 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.242023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.242026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.242198 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.242676 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.242679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.242681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.242952 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:52.243155 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.243833 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.244012 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
I20260807 11:16:52.245862 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.226047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45261 (local address 127.2.223.131:54871)
0807 11:16:52.226832 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.226872 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.228703 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.229564 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.229576 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.230362 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:52.231089 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.231108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.232478 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.232483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.234195 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.234214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.235010 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.235016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.235235 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.235874 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:16:52.235902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:52.238340 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:52.241363 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:52.241370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:52.241385 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:52.241845 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:52.242306 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:16:52.242310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:16:52.242312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:16:52.242557 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:16:52.243129 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:52.243136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:52.243481 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.243736 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.244304 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":288,"threads_started":1}
I20260807 11:16:52.247346 7802 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45261
I20260807 11:16:52.247637 7802 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:52.248282 7802 heartbeater.cc:511] Master 127.2.223.190:45261 requested a full tablet report, sending...
I20260807 11:16:52.249620 7325 ts_manager.cc:194] Registered new tserver with Master: 8c1960357c224fc2a1d850d27e36d031 (127.2.223.131:43395)
I20260807 11:16:52.250427 7325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:54871
I20260807 11:16:52.258638 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:16:52.276690 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.263652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.2.223.190:45261)
0807 11:16:52.263909 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.263915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.264239 (+ 324us) server_negotiation.cc:407] Connection header received
0807 11:16:52.264379 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.264383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.264449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.264530 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:52.265927 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.266693 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.268030 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.268443 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.268574 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.270997 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:52.271002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:52.271005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:52.271039 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:52.273909 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.274580 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.274584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.274587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.274724 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.275072 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.275076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.275079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.275492 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:52.275721 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.276021 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.276225 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:16:52.280984 7325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:52.288147 7325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:52.314363 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.301157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:43453 (local address 127.0.0.1:40898)
0807 11:16:52.302346 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.302384 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.302614 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.303705 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.303717 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.303759 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:52.304245 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.304261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.306836 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.306845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.309992 (+ 3147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.310013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.312967 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.312978 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.313287 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.313359 (+ 72us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.313525 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.313812 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":881,"threads_started":1}
I20260807 11:16:52.315562 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.301178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.2.223.130:43453)
0807 11:16:52.301906 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.301914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.302526 (+ 612us) server_negotiation.cc:407] Connection header received
0807 11:16:52.302846 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.302853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.303117 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.303343 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:52.304483 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.306618 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.310241 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.312342 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.312522 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.312802 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.314894 (+ 2092us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.315134 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":473,"threads_started":1}
I20260807 11:16:52.320691 7593 tablet_service.cc:1467] Processing CreateTablet for tablet 911885c7dda8462e91eb276b623fbf75 (DEFAULT_TABLE table=test-table [id=6829592b627d4411acf1117d8f0541e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:52.322144 7593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911885c7dda8462e91eb276b623fbf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:52.323681 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.304150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.2.223.129:37407)
0807 11:16:52.315264 (+ 11114us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.315271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.315297 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:52.315377 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.315383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.315710 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.316606 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:52.316791 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.318241 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.321025 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.322472 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.322630 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.322862 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.323154 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.323324 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10921,"thread_start_us":10733,"threads_started":1}
I20260807 11:16:52.323719 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.303494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37407 (local address 127.0.0.1:58970)
0807 11:16:52.314200 (+ 10706us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.314225 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.314392 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.315968 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.315974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.316001 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:52.316415 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.316429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.318420 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.318429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.320839 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.320860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.322726 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.322735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.322947 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.322983 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.323108 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.323251 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10515,"mutex_wait_us":8,"thread_start_us":11032,"threads_started":1}
I20260807 11:16:52.326169 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.302819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:43395 (local address 127.0.0.1:50448)
0807 11:16:52.303429 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.303448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.303576 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.318993 (+ 15417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.319001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.319036 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:52.319421 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.319428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.320790 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.320799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.322921 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.322941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.325331 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.325339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.325541 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.325579 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.325680 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.325821 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
I20260807 11:16:52.326318 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.303117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.2.223.131:43395)
0807 11:16:52.318054 (+ 14937us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.318062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.318099 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:16:52.318198 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.318208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.318440 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.318682 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:52.319638 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.320642 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.323071 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.325116 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.325261 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.325476 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.325775 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.325944 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14777,"thread_start_us":14608,"threads_started":1}
I20260807 11:16:52.327864 7453 tablet_service.cc:1467] Processing CreateTablet for tablet 911885c7dda8462e91eb276b623fbf75 (DEFAULT_TABLE table=test-table [id=6829592b627d4411acf1117d8f0541e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:52.329067 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911885c7dda8462e91eb276b623fbf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:52.330228 7734 tablet_service.cc:1467] Processing CreateTablet for tablet 911885c7dda8462e91eb276b623fbf75 (DEFAULT_TABLE table=test-table [id=6829592b627d4411acf1117d8f0541e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:52.331576 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911885c7dda8462e91eb276b623fbf75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:52.338402 7823 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Bootstrap starting.
I20260807 11:16:52.342934 7823 tablet_bootstrap.cc:654] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:52.342993 7824 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Bootstrap starting.
I20260807 11:16:52.344200 7823 log.cc:824] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:52.346679 7824 tablet_bootstrap.cc:654] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:52.346932 7823 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: No bootstrap required, opened a new log
I20260807 11:16:52.347219 7823 ts_tablet_manager.cc:1397] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260807 11:16:52.348063 7824 log.cc:824] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:52.348304 7826 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Bootstrap starting.
I20260807 11:16:52.351781 7826 tablet_bootstrap.cc:654] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:52.352037 7824 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: No bootstrap required, opened a new log
I20260807 11:16:52.352362 7824 ts_tablet_manager.cc:1397] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 11:16:52.352952 7826 log.cc:824] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:52.356172 7826 tablet_bootstrap.cc:492] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: No bootstrap required, opened a new log
I20260807 11:16:52.356436 7826 ts_tablet_manager.cc:1397] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:16:52.357992 7823 raft_consensus.cc:348] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.358455 7823 raft_consensus.cc:374] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:52.358603 7823 raft_consensus.cc:729] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a90a9b8251410ab791cca5a34b7d9d, State: Initialized, Role: FOLLOWER
I20260807 11:16:52.359217 7823 consensus_queue.cc:260] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.360395 7662 heartbeater.cc:503] Master 127.2.223.190:45261 was elected leader, sending a full tablet report...
I20260807 11:16:52.360967 7823 ts_tablet_manager.cc:1428] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 11:16:52.362226 7824 raft_consensus.cc:348] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.362747 7824 raft_consensus.cc:374] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:52.362879 7824 raft_consensus.cc:729] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c94dd7b4cdb4be381d1df261b0fb4e6, State: Initialized, Role: FOLLOWER
I20260807 11:16:52.363722 7824 consensus_queue.cc:260] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.367725 7826 raft_consensus.cc:348] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.367982 7824 ts_tablet_manager.cc:1428] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260807 11:16:52.368060 7826 raft_consensus.cc:374] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:52.368247 7826 raft_consensus.cc:729] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1960357c224fc2a1d850d27e36d031, State: Initialized, Role: FOLLOWER
I20260807 11:16:52.368831 7826 consensus_queue.cc:260] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.370030 7802 heartbeater.cc:503] Master 127.2.223.190:45261 was elected leader, sending a full tablet report...
I20260807 11:16:52.370424 7826 ts_tablet_manager.cc:1428] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260807 11:16:52.371342 7522 tablet.cc:2369] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:16:52.402940 7663 tablet.cc:2369] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:52.416225 7831 raft_consensus.cc:482] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:52.416570 7831 raft_consensus.cc:504] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.418424 7831 leader_election.cc:288] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a90a9b8251410ab791cca5a34b7d9d (127.2.223.130:43453), 1c94dd7b4cdb4be381d1df261b0fb4e6 (127.2.223.129:37407)
I20260807 11:16:52.424705 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.418794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:43453 (local address 127.2.223.131:39999)
0807 11:16:52.419006 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.419027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.419155 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.419504 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.419509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.419546 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:52.419777 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.419792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.421087 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.421090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.422809 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.422824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.424182 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.424186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.424323 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.424346 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.424420 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.424506 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 11:16:52.424853 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.418930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:39999 (local address 127.2.223.130:43453)
0807 11:16:52.419133 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.419138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.419182 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:16:52.419266 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.419272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.419348 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.419486 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:52.419962 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.420939 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.422987 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.424040 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.424195 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.424270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.424472 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.424606 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:16:52.425947 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.419237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37407 (local address 127.2.223.131:60301)
0807 11:16:52.420075 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:52.420090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:52.420186 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:52.420486 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:52.420489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:52.420507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:16:52.420802 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.420810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.421870 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.421876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.423816 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:52.423828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.425398 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.425407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:52.425540 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.425561 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 11:16:52.425640 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 11:16:52.425720 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":548,"threads_started":1}
I20260807 11:16:52.426105 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.419540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:60301 (local address 127.2.223.129:37407)
0807 11:16:52.419801 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.419809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.420181 (+ 372us) server_negotiation.cc:407] Connection header received
0807 11:16:52.420282 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.420288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.420369 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.420470 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:16:52.420954 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.421715 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.423966 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.425205 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.425329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.425375 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.425703 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.425838 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 11:16:52.425935 7613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911885c7dda8462e91eb276b623fbf75" candidate_uuid: "8c1960357c224fc2a1d850d27e36d031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a90a9b8251410ab791cca5a34b7d9d" is_pre_election: true
I20260807 11:16:52.426571 7613 raft_consensus.cc:2454] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1960357c224fc2a1d850d27e36d031 in term 0.
I20260807 11:16:52.427276 7684 leader_election.cc:302] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a90a9b8251410ab791cca5a34b7d9d, 8c1960357c224fc2a1d850d27e36d031; no voters:
I20260807 11:16:52.427201 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911885c7dda8462e91eb276b623fbf75" candidate_uuid: "8c1960357c224fc2a1d850d27e36d031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" is_pre_election: true
I20260807 11:16:52.427683 7831 raft_consensus.cc:2781] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:52.427713 7474 raft_consensus.cc:2454] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1960357c224fc2a1d850d27e36d031 in term 0.
I20260807 11:16:52.427865 7831 raft_consensus.cc:482] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:52.427983 7831 raft_consensus.cc:3037] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:52.430529 7831 raft_consensus.cc:504] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.431238 7831 leader_election.cc:288] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [CANDIDATE]: Term 1 election: Requested vote from peers 37a90a9b8251410ab791cca5a34b7d9d (127.2.223.130:43453), 1c94dd7b4cdb4be381d1df261b0fb4e6 (127.2.223.129:37407)
I20260807 11:16:52.431664 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911885c7dda8462e91eb276b623fbf75" candidate_uuid: "8c1960357c224fc2a1d850d27e36d031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6"
I20260807 11:16:52.431674 7613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911885c7dda8462e91eb276b623fbf75" candidate_uuid: "8c1960357c224fc2a1d850d27e36d031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a90a9b8251410ab791cca5a34b7d9d"
I20260807 11:16:52.431854 7474 raft_consensus.cc:3037] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:52.431864 7613 raft_consensus.cc:3037] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:52.434013 7474 raft_consensus.cc:2454] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1960357c224fc2a1d850d27e36d031 in term 1.
I20260807 11:16:52.434067 7613 raft_consensus.cc:2454] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1960357c224fc2a1d850d27e36d031 in term 1.
I20260807 11:16:52.434427 7684 leader_election.cc:302] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c94dd7b4cdb4be381d1df261b0fb4e6, 8c1960357c224fc2a1d850d27e36d031; no voters:
I20260807 11:16:52.434782 7831 raft_consensus.cc:2781] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:52.435257 7831 raft_consensus.cc:686] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1960357c224fc2a1d850d27e36d031, State: Running, Role: LEADER
I20260807 11:16:52.435755 7831 consensus_queue.cc:237] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } }
I20260807 11:16:52.440595 7323 catalog_manager.cc:5495] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1960357c224fc2a1d850d27e36d031 (127.2.223.131). New cstate: current_term: 1 leader_uuid: "8c1960357c224fc2a1d850d27e36d031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c1960357c224fc2a1d850d27e36d031" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 } health_report { overall_health: UNKNOWN } } }
I20260807 11:16:52.465663 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.460258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.2.223.131:43395)
0807 11:16:52.460522 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.460528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.460693 (+ 165us) server_negotiation.cc:407] Connection header received
0807 11:16:52.460894 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.460897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.460981 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.461108 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:16:52.461440 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.462236 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.463745 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.464133 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.464227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.464645 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:16:52.465165 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:16:52.465274 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.465400 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.465464 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:16:52.473515 7474 raft_consensus.cc:1261] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 1 FOLLOWER]: Refusing update from remote peer 8c1960357c224fc2a1d850d27e36d031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:52.473649 7613 raft_consensus.cc:1261] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 1 FOLLOWER]: Refusing update from remote peer 8c1960357c224fc2a1d850d27e36d031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 11:16:52.474403 7803 tablet.cc:2369] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:16:52.474699 7833 consensus_queue.cc:1035] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a90a9b8251410ab791cca5a34b7d9d" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:52.475414 7831 consensus_queue.cc:1035] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c94dd7b4cdb4be381d1df261b0fb4e6" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:52.486763 7836 mvcc.cc:205] Tried to move back new op lower bound from 7315871385482207232 to 7315871385340866560. Current Snapshot: MvccSnapshot[applied={T|T < 7315871385482207232}]
I20260807 11:16:52.489478 7837 mvcc.cc:205] Tried to move back new op lower bound from 7315871385482207232 to 7315871385340866560. Current Snapshot: MvccSnapshot[applied={T|T < 7315871385482207232}]
I20260807 11:16:52.500198 7322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:52.500497 7322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:52.504459 7322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65efb51fab03400d99e1968ac3402f59: Sending DeleteTablet for 3 replicas of tablet 911885c7dda8462e91eb276b623fbf75
I20260807 11:16:52.505823 7734 tablet_service.cc:1514] Processing DeleteTablet for tablet 911885c7dda8462e91eb276b623fbf75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:52 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50448
I20260807 11:16:52.505967 7593 tablet_service.cc:1514] Processing DeleteTablet for tablet 911885c7dda8462e91eb276b623fbf75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:52 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:40898
I20260807 11:16:52.506610 7453 tablet_service.cc:1514] Processing DeleteTablet for tablet 911885c7dda8462e91eb276b623fbf75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:52 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:58970
I20260807 11:16:52.508744 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7389
I20260807 11:16:52.517526 7848 tablet_replica.cc:330] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: stopping tablet replica
I20260807 11:16:52.518267 7848 raft_consensus.cc:2229] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:52.519071 7848 raft_consensus.cc:2258] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:52.519793 7847 tablet_replica.cc:330] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: stopping tablet replica
I20260807 11:16:52.520711 7847 raft_consensus.cc:2229] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:52.521605 7847 raft_consensus.cc:2258] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:52.523394 7848 ts_tablet_manager.cc:1905] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:52.532261 7848 ts_tablet_manager.cc:1918] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:52.532528 7848 log.cc:1196] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/911885c7dda8462e91eb276b623fbf75
I20260807 11:16:52.532902 7847 ts_tablet_manager.cc:1905] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:52.533218 7848 ts_tablet_manager.cc:1939] T 911885c7dda8462e91eb276b623fbf75 P 37a90a9b8251410ab791cca5a34b7d9d: Deleting consensus metadata
I20260807 11:16:52.535744 7308 catalog_manager.cc:4841] TS 37a90a9b8251410ab791cca5a34b7d9d (127.2.223.130:43453): tablet 911885c7dda8462e91eb276b623fbf75 (table test-table [id=6829592b627d4411acf1117d8f0541e9]) successfully deleted
I20260807 11:16:52.539511 7849 tablet_replica.cc:330] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: stopping tablet replica
I20260807 11:16:52.540067 7849 raft_consensus.cc:2229] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:52.540693 7849 raft_consensus.cc:2258] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:52.543269 7849 ts_tablet_manager.cc:1905] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:52.543491 7847 ts_tablet_manager.cc:1918] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:52.543741 7847 log.cc:1196] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/911885c7dda8462e91eb276b623fbf75
I20260807 11:16:52.544394 7847 ts_tablet_manager.cc:1939] T 911885c7dda8462e91eb276b623fbf75 P 8c1960357c224fc2a1d850d27e36d031: Deleting consensus metadata
I20260807 11:16:52.546519 7308 catalog_manager.cc:4841] TS 8c1960357c224fc2a1d850d27e36d031 (127.2.223.131:43395): tablet 911885c7dda8462e91eb276b623fbf75 (table test-table [id=6829592b627d4411acf1117d8f0541e9]) successfully deleted
Aug 07 11:16:52 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:16:52.550559 7849 ts_tablet_manager.cc:1918] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:52.550784 7849 log.cc:1196] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/911885c7dda8462e91eb276b623fbf75
I20260807 11:16:52.551384 7849 ts_tablet_manager.cc:1939] T 911885c7dda8462e91eb276b623fbf75 P 1c94dd7b4cdb4be381d1df261b0fb4e6: Deleting consensus metadata
I20260807 11:16:52.553258 7308 catalog_manager.cc:4841] TS 1c94dd7b4cdb4be381d1df261b0fb4e6 (127.2.223.129:37407): tablet 911885c7dda8462e91eb276b623fbf75 (table test-table [id=6829592b627d4411acf1117d8f0541e9]) successfully deleted
I20260807 11:16:52.557379 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.510589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.2.223.129:37407)
0807 11:16:52.510827 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.510837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.538560 (+ 27723us) server_negotiation.cc:407] Connection header received
0807 11:16:52.539061 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.539070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.539182 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.539307 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:52.542928 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.544097 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.545936 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.546589 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.546685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.550792 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:52.550835 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:52.550855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:52.550906 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:52.554454 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.555195 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.555205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.555209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.555410 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.555823 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.555828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.555831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.556198 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:52.556543 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.556781 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.557000 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:16:52.558132 7516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:52.649475 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7389
I20260807 11:16:52.679733 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7527
Aug 07 11:16:52 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:16:52.697978 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.680650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.2.223.130:43453)
0807 11:16:52.680844 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.680850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.680915 (+ 65us) server_negotiation.cc:407] Connection header received
0807 11:16:52.681150 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.681156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.681234 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.681364 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:52.683051 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.684310 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.686229 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.686911 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.687029 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.691960 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:52.691971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:52.691986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:52.692028 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:52.695141 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.695805 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.695819 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.695821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.695948 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.696336 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.696342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.696345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.696744 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:52.697065 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.697379 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.697610 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:16:52.699086 7656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:52.790726 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7527
I20260807 11:16:52.819833 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7668
Aug 07 11:16:52 dist-test-slave-5l3k krb5kdc[7280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:16:52.837244 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:52.820764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.2.223.131:43395)
0807 11:16:52.820971 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:52.820979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:52.821003 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:16:52.821127 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:52.821131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:52.821207 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:52.821329 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:52.822692 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.823526 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.824778 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:52.825395 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:52.825540 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:52.830717 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:52.830729 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:52.830747 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:16:52.830785 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:52.833952 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.834751 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.834762 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.834764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.834919 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:16:52.835402 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:16:52.835409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:16:52.835412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:16:52.835929 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:52.836272 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:52.836555 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 11:16:52.836815 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:16:52.838135 7797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:52.968214 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7668
I20260807 11:16:52.994622 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7293
I20260807 11:16:52.995824 7356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:53.108505 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7293
2026-08-07T11:16:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4426 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T11:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:53Z Disabled control of system clock
I20260807 11:16:53.187927 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33307
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 11:16:53.518775 7858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:53.519071 7858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:53.519263 7858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:53.526615 7858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:53.526719 7858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:53.526811 7858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:53.526909 7858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:53.526988 7858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:16:53.536273 7858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:53.537603 7858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:53.539446 7858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:53.551445 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:53.551479 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:53.551388 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:53.552853 7858 server_base.cc:1034] running on GCE node
I20260807 11:16:53.553587 7858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:53.555059 7858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:53.556268 7858 hybrid_clock.cc:648] HybridClock initialized: now 1786101413556233 us; error 25 us; skew 500 ppm
I20260807 11:16:53.556651 7858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:53.558914 7858 webserver.cc:460] Webserver started at http://127.2.223.190:39473/ using document root <none> and password file <none>
I20260807 11:16:53.559583 7858 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:53.559674 7858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:53.559947 7858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:53.562834 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "ada00da8566f497a9187b72cb7dda02f"
format_stamp: "Formatted at 2026-08-07 11:16:53 on dist-test-slave-5l3k"
server_key: "4bea5de5252da4b4bef9b587d1f42ef0"
server_key_iv: "8c4c02081e163250b6c7524bcb66f54f"
server_key_version: "encryptionkey@0"
I20260807 11:16:53.563580 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "ada00da8566f497a9187b72cb7dda02f"
format_stamp: "Formatted at 2026-08-07 11:16:53 on dist-test-slave-5l3k"
server_key: "4bea5de5252da4b4bef9b587d1f42ef0"
server_key_iv: "8c4c02081e163250b6c7524bcb66f54f"
server_key_version: "encryptionkey@0"
I20260807 11:16:53.568573 7858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:53.572049 7873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:53.573467 7858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:16:53.573606 7858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "ada00da8566f497a9187b72cb7dda02f"
format_stamp: "Formatted at 2026-08-07 11:16:53 on dist-test-slave-5l3k"
server_key: "4bea5de5252da4b4bef9b587d1f42ef0"
server_key_iv: "8c4c02081e163250b6c7524bcb66f54f"
server_key_version: "encryptionkey@0"
I20260807 11:16:53.573784 7858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:53.579648 7858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:53.580363 7858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:53.580611 7858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:53.599071 7858 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33307
I20260807 11:16:53.599160 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33307 every 8 connection(s)
I20260807 11:16:53.600414 7858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:53.604288 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:53.607838 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7858
I20260807 11:16:53.608023 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:53.608880 2942 external_mini_cluster.cc:1428] Setting key 61c077cf0f078e9e94d39fadfbde04da
I20260807 11:16:53.617465 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:53.611093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.2.223.190:33307)
0807 11:16:53.612073 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:53.612084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:53.612126 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:16:53.613769 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:53.613792 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:53.614225 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:53.614687 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:53.615855 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:53.615861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:53.615873 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:53.615886 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:53.615942 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:53.615957 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:53.616194 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:53.616341 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 11:16:53.616595 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":346,"threads_started":1}
I20260807 11:16:53.617558 7928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: Bootstrap starting.
I20260807 11:16:53.621054 7928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:53.622026 7928 log.cc:824] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:53.624626 7928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: No bootstrap required, opened a new log
I20260807 11:16:53.631901 7928 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } }
I20260807 11:16:53.632277 7928 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:53.632386 7928 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada00da8566f497a9187b72cb7dda02f, State: Initialized, Role: FOLLOWER
I20260807 11:16:53.632992 7928 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } }
I20260807 11:16:53.633327 7928 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:53.633500 7928 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:53.633703 7928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:53.635902 7928 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } }
I20260807 11:16:53.636402 7928 leader_election.cc:302] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada00da8566f497a9187b72cb7dda02f; no voters:
I20260807 11:16:53.636956 7928 leader_election.cc:288] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:53.637085 7934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:53.637903 7934 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [term 1 LEADER]: Becoming Leader. State: Replica: ada00da8566f497a9187b72cb7dda02f, State: Running, Role: LEADER
I20260807 11:16:53.638559 7934 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } }
I20260807 11:16:53.639219 7928 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:53.641356 7936 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada00da8566f497a9187b72cb7dda02f. Latest consensus state: current_term: 1 leader_uuid: "ada00da8566f497a9187b72cb7dda02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } } }
I20260807 11:16:53.641799 7936 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:53.641961 7935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada00da8566f497a9187b72cb7dda02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada00da8566f497a9187b72cb7dda02f" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33307 } } }
I20260807 11:16:53.642524 7935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:53.647693 7939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:53.654134 7939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:53.664391 7939 catalog_manager.cc:1347] Generated new cluster ID: f12f6ad12e364879b1fcf606cfdacf0e
I20260807 11:16:53.664517 7939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:53.679028 7939 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:53.679437 7939 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:53.685770 7939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: Generated new TSK 0
I20260807 11:16:53.686589 7939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:16:53.691255 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--builtin_ntp_servers=127.2.223.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 11:16:54.029728 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:54.030004 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:54.030154 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:54.037446 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:54.037536 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:54.037699 7954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:16:54.046913 7954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:54.048151 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:54.050060 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:54.063723 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.063820 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.063845 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:54.065326 7954 server_base.cc:1034] running on GCE node
I20260807 11:16:54.066181 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:54.067341 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:54.068569 7954 hybrid_clock.cc:648] HybridClock initialized: now 1786101414068496 us; error 62 us; skew 500 ppm
I20260807 11:16:54.068958 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:54.071744 7954 webserver.cc:460] Webserver started at http://127.2.223.129:34045/ using document root <none> and password file <none>
I20260807 11:16:54.072546 7954 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:54.072690 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:54.073100 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:54.076635 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "b54c2a3db6eb4d1593214c67f5286e50"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "efd2ce485954e10639a21d6169ca7b1a"
server_key_iv: "4d56d483128ebab702465d69750b2b31"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.077672 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "b54c2a3db6eb4d1593214c67f5286e50"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "efd2ce485954e10639a21d6169ca7b1a"
server_key_iv: "4d56d483128ebab702465d69750b2b31"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.083822 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:16:54.087549 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:54.088860 7954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:16:54.089056 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "b54c2a3db6eb4d1593214c67f5286e50"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "efd2ce485954e10639a21d6169ca7b1a"
server_key_iv: "4d56d483128ebab702465d69750b2b31"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.089289 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:54.094887 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:54.095664 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:54.095930 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:54.096824 7954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:54.098196 7954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:54.098282 7954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:54.098433 7954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:54.098578 7954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 11:16:54.133252 7954 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:33365
I20260807 11:16:54.133354 8085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:33365 every 8 connection(s)
I20260807 11:16:54.134663 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:16:54.135666 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 7954
I20260807 11:16:54.135857 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:16:54.136598 2942 external_mini_cluster.cc:1428] Setting key c5f8e462737ecb2c1388374b43e05130
I20260807 11:16:54.139639 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--builtin_ntp_servers=127.2.223.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 11:16:54.144048 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:54.137402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:47893 (local address 127.2.223.190:33307)
0807 11:16:54.138759 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:54.138769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:54.139880 (+ 1111us) server_negotiation.cc:407] Connection header received
0807 11:16:54.141633 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:54.141638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:54.141709 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:54.141821 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:54.143212 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:54.143218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:54.143222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:54.143242 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:54.143297 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:54.143308 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:54.143436 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:54.143663 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 11:16:54.143733 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":1021,"threads_started":1}
I20260807 11:16:54.144953 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:54.138094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33307 (local address 127.2.223.129:47893)
0807 11:16:54.139708 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:54.139753 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:54.141361 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:54.142056 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:54.142084 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:54.142172 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:54.142917 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:54.142919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:54.142957 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:54.143041 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:54.143407 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:54.143419 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:54.143427 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 11:16:54.143757 (+ 330us) client_negotiation.cc:240] Negotiation successful
0807 11:16:54.144003 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":818,"threads_started":1}
I20260807 11:16:54.146552 8086 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33307
I20260807 11:16:54.146806 8086 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:54.147523 8086 heartbeater.cc:511] Master 127.2.223.190:33307 requested a full tablet report, sending...
I20260807 11:16:54.149868 7892 ts_manager.cc:194] Registered new tserver with Master: b54c2a3db6eb4d1593214c67f5286e50 (127.2.223.129:33365)
W20260807 11:16:54.476802 8090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:54.477098 8090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:54.477283 8090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:54.484700 8090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:54.484813 8090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:54.484998 8090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:16:54.494371 8090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:54.495729 8090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:54.497618 8090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:54.511870 8096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.511822 8099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.511924 8095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:54.516047 8090 server_base.cc:1034] running on GCE node
I20260807 11:16:54.516736 8090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:54.517741 8090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:54.518955 8090 hybrid_clock.cc:648] HybridClock initialized: now 1786101414518871 us; error 71 us; skew 500 ppm
I20260807 11:16:54.519317 8090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:54.521749 8090 webserver.cc:460] Webserver started at http://127.2.223.130:44311/ using document root <none> and password file <none>
I20260807 11:16:54.522455 8090 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:54.522629 8090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:54.522996 8090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:54.525962 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "ba7bde098c2b46e98a0b8709d2216bfc"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "7ded699dd3252370c864ce330ceaa658"
server_key_iv: "766795513d367762168e68db6c67a224"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.526849 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "ba7bde098c2b46e98a0b8709d2216bfc"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "7ded699dd3252370c864ce330ceaa658"
server_key_iv: "766795513d367762168e68db6c67a224"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.532189 8090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:54.535578 8105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:54.536870 8090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:16:54.537043 8090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "ba7bde098c2b46e98a0b8709d2216bfc"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "7ded699dd3252370c864ce330ceaa658"
server_key_iv: "766795513d367762168e68db6c67a224"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.537297 8090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:54.543020 8090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:54.543812 8090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:54.544104 8090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:54.545037 8090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:54.546432 8090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:54.546622 8090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:54.546741 8090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:54.546818 8090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:54.582813 8090 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:46011
I20260807 11:16:54.583034 8222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:46011 every 8 connection(s)
I20260807 11:16:54.584587 8090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:16:54.593133 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:54.587314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49625 (local address 127.2.223.190:33307)
0807 11:16:54.587503 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:54.587507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:54.588916 (+ 1409us) server_negotiation.cc:407] Connection header received
0807 11:16:54.590731 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:54.590737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:54.590811 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:54.590928 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:54.592287 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:54.592293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:54.592297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:54.592328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:54.592362 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:54.592373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:54.592511 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:54.592776 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 11:16:54.592830 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:16:54.594059 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:54.587847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33307 (local address 127.2.223.130:49625)
0807 11:16:54.588755 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:54.588798 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:54.590382 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:54.591163 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:54.591170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:54.591259 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:54.591985 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:54.591988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:54.592024 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:54.592111 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:54.592503 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:54.592512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:54.592519 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:16:54.592824 (+ 305us) client_negotiation.cc:240] Negotiation successful
0807 11:16:54.593034 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260807 11:16:54.594966 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8090
I20260807 11:16:54.595134 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:16:54.595530 8223 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33307
I20260807 11:16:54.595574 2942 external_mini_cluster.cc:1428] Setting key 57c743b7f90f095ae24ee41926c08c72
I20260807 11:16:54.595824 8223 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:54.596487 8223 heartbeater.cc:511] Master 127.2.223.190:33307 requested a full tablet report, sending...
I20260807 11:16:54.597992 7892 ts_manager.cc:194] Registered new tserver with Master: ba7bde098c2b46e98a0b8709d2216bfc (127.2.223.130:46011)
I20260807 11:16:54.598564 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--builtin_ntp_servers=127.2.223.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 11:16:54.949970 8226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:54.950299 8226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:54.950505 8226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:54.957687 8226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:54.957800 8226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:54.957912 8226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:16:54.966882 8226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:54.968097 8226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:54.969847 8226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:54.983673 8231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.983700 8232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:54.983634 8234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:54.984728 8226 server_base.cc:1034] running on GCE node
I20260807 11:16:54.985322 8226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:54.986243 8226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:54.987470 8226 hybrid_clock.cc:648] HybridClock initialized: now 1786101414987387 us; error 68 us; skew 500 ppm
I20260807 11:16:54.987855 8226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:54.990461 8226 webserver.cc:460] Webserver started at http://127.2.223.131:40813/ using document root <none> and password file <none>
I20260807 11:16:54.991245 8226 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:54.991377 8226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:54.991714 8226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:54.994917 8226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "783f004df8074f949239090a4e0dae47"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "c1e352de8040e1e493ab3e65cdd57932"
server_key_iv: "01eb25378f6f5feedd92359ad1364a9a"
server_key_version: "encryptionkey@0"
I20260807 11:16:54.995765 8226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "783f004df8074f949239090a4e0dae47"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "c1e352de8040e1e493ab3e65cdd57932"
server_key_iv: "01eb25378f6f5feedd92359ad1364a9a"
server_key_version: "encryptionkey@0"
I20260807 11:16:55.001057 8226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:16:55.004662 8240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:55.006044 8226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:55.006254 8226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "783f004df8074f949239090a4e0dae47"
format_stamp: "Formatted at 2026-08-07 11:16:54 on dist-test-slave-5l3k"
server_key: "c1e352de8040e1e493ab3e65cdd57932"
server_key_iv: "01eb25378f6f5feedd92359ad1364a9a"
server_key_version: "encryptionkey@0"
I20260807 11:16:55.006510 8226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:55.012205 8226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:55.013041 8226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:55.013339 8226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:55.014312 8226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:16:55.015818 8226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:16:55.015951 8226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:55.016084 8226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:16:55.016188 8226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:55.050959 8226 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:42851
I20260807 11:16:55.051067 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:42851 every 8 connection(s)
I20260807 11:16:55.052357 8226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:16:55.054111 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8226
I20260807 11:16:55.054279 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:16:55.054916 2942 external_mini_cluster.cc:1428] Setting key ebc978f4aa6acbceb981144fe7ff5318
I20260807 11:16:55.062559 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.056174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:52273 (local address 127.2.223.190:33307)
0807 11:16:55.056391 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.056398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.058190 (+ 1792us) server_negotiation.cc:407] Connection header received
0807 11:16:55.060448 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.060452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.060508 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.060596 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.061891 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.061896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.061899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.061915 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.061949 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.061960 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.062062 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.062237 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.062286 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:16:55.063500 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.056856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33307 (local address 127.2.223.131:52273)
0807 11:16:55.058006 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.058047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.060185 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.060873 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.060882 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.060979 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.061605 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.061608 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.061645 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.061730 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.062055 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.062063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.062068 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.062282 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.062547 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":377,"threads_started":1}
I20260807 11:16:55.064957 8359 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33307
I20260807 11:16:55.065297 8359 heartbeater.cc:464] Registering TS with master...
I20260807 11:16:55.065884 8359 heartbeater.cc:511] Master 127.2.223.190:33307 requested a full tablet report, sending...
I20260807 11:16:55.067332 7892 ts_manager.cc:194] Registered new tserver with Master: 783f004df8074f949239090a4e0dae47 (127.2.223.131:42851)
I20260807 11:16:55.071075 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:16:55.078044 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.075511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.2.223.190:33307)
0807 11:16:55.075732 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.075740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.076177 (+ 437us) server_negotiation.cc:407] Connection header received
0807 11:16:55.076306 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.076310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.076362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.076442 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.077412 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.077416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.077419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.077433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.077477 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.077489 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.077584 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.077669 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.077714 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:16:55.081868 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:16:55.088896 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:16:55.116848 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.102982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.2.223.130:46011)
0807 11:16:55.112619 (+ 9637us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.112629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.113999 (+ 1370us) server_negotiation.cc:407] Connection header received
0807 11:16:55.114082 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.114092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.114335 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.114530 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.115954 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.115967 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.115978 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.115993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.116042 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.116061 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.116229 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.116393 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.116500 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9449,"thread_start_us":9324,"threads_started":1}
I20260807 11:16:55.117007 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.102509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:46011 (local address 127.0.0.1:50848)
0807 11:16:55.113740 (+ 11231us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.113778 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.113947 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.114540 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.114548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.114649 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.115617 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.115621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.115651 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.115749 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.116178 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.116190 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.116198 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.116398 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.116494 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10934,"mutex_wait_us":156,"spinlock_wait_cycles":23936,"thread_start_us":9740,"threads_started":1}
I20260807 11:16:55.117123 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.102957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.2.223.129:33365)
0807 11:16:55.112140 (+ 9183us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.112150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.113929 (+ 1779us) server_negotiation.cc:407] Connection header received
0807 11:16:55.114149 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.114181 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.114410 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.114589 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.116154 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.116163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.116176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.116194 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.116250 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.116273 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.116410 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.116635 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.116734 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9030,"thread_start_us":8965,"threads_started":1}
I20260807 11:16:55.117624 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.102509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45360)
0807 11:16:55.113722 (+ 11213us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.113778 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.113947 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.115126 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.115131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.115154 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.115737 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.115740 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.115752 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.115804 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.116463 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.116467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.116472 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.116633 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.116689 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10501,"spinlock_wait_cycles":1059584,"thread_start_us":10752,"threads_started":1}
I20260807 11:16:55.118673 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.101976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:42851 (local address 127.0.0.1:55282)
0807 11:16:55.113724 (+ 11748us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.113782 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.114032 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.115724 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.115732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.115758 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.117267 (+ 1509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.117272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.117295 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.117340 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.117948 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.117952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.117957 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.118106 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.118158 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10626,"thread_start_us":10779,"threads_started":1}
I20260807 11:16:55.119483 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.102150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.2.223.131:42851)
0807 11:16:55.111905 (+ 9755us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.111926 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.113928 (+ 2002us) server_negotiation.cc:407] Connection header received
0807 11:16:55.114253 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.114260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.114410 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.114594 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.117498 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.117507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.117519 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.117544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.117618 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.117639 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.117884 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.118976 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.119085 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9552,"thread_start_us":9374,"threads_started":1}
I20260807 11:16:55.121552 8155 tablet_service.cc:1467] Processing CreateTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 (DEFAULT_TABLE table=test-table [id=d73f63e6d5184f1d98111da8df468c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:55.122112 8018 tablet_service.cc:1467] Processing CreateTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 (DEFAULT_TABLE table=test-table [id=d73f63e6d5184f1d98111da8df468c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:55.122198 8290 tablet_service.cc:1467] Processing CreateTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 (DEFAULT_TABLE table=test-table [id=d73f63e6d5184f1d98111da8df468c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:16:55.123234 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:55.123620 8290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:55.123709 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:55.143833 8377 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Bootstrap starting.
I20260807 11:16:55.144462 8379 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Bootstrap starting.
I20260807 11:16:55.146894 8378 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: Bootstrap starting.
I20260807 11:16:55.147528 8377 tablet_bootstrap.cc:654] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:55.148866 8377 log.cc:824] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:55.150380 8378 tablet_bootstrap.cc:654] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:55.151742 8378 log.cc:824] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:55.152227 8377 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: No bootstrap required, opened a new log
I20260807 11:16:55.152567 8377 ts_tablet_manager.cc:1397] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260807 11:16:55.152626 8086 heartbeater.cc:503] Master 127.2.223.190:33307 was elected leader, sending a full tablet report...
I20260807 11:16:55.155753 8378 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: No bootstrap required, opened a new log
I20260807 11:16:55.156019 8378 ts_tablet_manager.cc:1397] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:16:55.156523 8379 tablet_bootstrap.cc:654] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:55.157857 8379 log.cc:824] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:55.161003 8379 tablet_bootstrap.cc:492] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: No bootstrap required, opened a new log
I20260807 11:16:55.161289 8379 ts_tablet_manager.cc:1397] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20260807 11:16:55.162910 8377 raft_consensus.cc:348] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.163451 8377 raft_consensus.cc:374] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:55.163615 8377 raft_consensus.cc:729] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f004df8074f949239090a4e0dae47, State: Initialized, Role: FOLLOWER
I20260807 11:16:55.164176 8378 raft_consensus.cc:348] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.164296 8377 consensus_queue.cc:260] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.164511 8378 raft_consensus.cc:374] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:55.164655 8378 raft_consensus.cc:729] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54c2a3db6eb4d1593214c67f5286e50, State: Initialized, Role: FOLLOWER
I20260807 11:16:55.165220 8378 consensus_queue.cc:260] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.167039 8378 ts_tablet_manager.cc:1428] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 11:16:55.171161 8359 heartbeater.cc:503] Master 127.2.223.190:33307 was elected leader, sending a full tablet report...
I20260807 11:16:55.171845 8377 ts_tablet_manager.cc:1428] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260807 11:16:55.173195 8379 raft_consensus.cc:348] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.173631 8379 raft_consensus.cc:374] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:55.173745 8379 raft_consensus.cc:729] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7bde098c2b46e98a0b8709d2216bfc, State: Initialized, Role: FOLLOWER
I20260807 11:16:55.174372 8379 consensus_queue.cc:260] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.175529 8223 heartbeater.cc:503] Master 127.2.223.190:33307 was elected leader, sending a full tablet report...
I20260807 11:16:55.176247 8379 ts_tablet_manager.cc:1428] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 11:16:55.196656 8383 raft_consensus.cc:482] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:16:55.196944 8383 raft_consensus.cc:504] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.198709 8383 leader_election.cc:288] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba7bde098c2b46e98a0b8709d2216bfc (127.2.223.130:46011), b54c2a3db6eb4d1593214c67f5286e50 (127.2.223.129:33365)
I20260807 11:16:55.201401 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.198866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:46011 (local address 127.2.223.131:46807)
0807 11:16:55.199135 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.199155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.199265 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.199784 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.199791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.199811 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.200555 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.200563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.200593 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.200643 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.200965 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.200970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.200974 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.201081 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.201140 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260807 11:16:55.201421 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.199079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:46807 (local address 127.2.223.130:46011)
0807 11:16:55.199270 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.199279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.199305 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:16:55.199399 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.199404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.199470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.199570 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.200806 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.200811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.200814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.200830 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.200860 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.200872 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.200994 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.201076 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.201162 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:16:55.202159 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.199486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:47081 (local address 127.2.223.129:33365)
0807 11:16:55.199669 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.199675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.200096 (+ 421us) server_negotiation.cc:407] Connection header received
0807 11:16:55.200258 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.200262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.200312 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.200442 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.201379 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.201385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.201389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.201417 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.201449 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.201461 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.201609 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.201825 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.201900 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:16:55.202291 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.199135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.2.223.131:47081)
0807 11:16:55.199965 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.199980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:55.200102 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:55.200483 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:55.200486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:55.200508 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:16:55.201164 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:16:55.201168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:16:55.201181 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:16:55.201218 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:16:55.201662 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:16:55.201667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:16:55.201672 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:16:55.201807 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 11:16:55.201860 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":461,"threads_started":1}
I20260807 11:16:55.202678 8175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d2b9f9a7f848529b6bb7f4d2e7de83" candidate_uuid: "783f004df8074f949239090a4e0dae47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" is_pre_election: true
I20260807 11:16:55.203289 8175 raft_consensus.cc:2454] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f004df8074f949239090a4e0dae47 in term 0.
I20260807 11:16:55.203410 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d2b9f9a7f848529b6bb7f4d2e7de83" candidate_uuid: "783f004df8074f949239090a4e0dae47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54c2a3db6eb4d1593214c67f5286e50" is_pre_election: true
I20260807 11:16:55.203994 8039 raft_consensus.cc:2454] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f004df8074f949239090a4e0dae47 in term 0.
I20260807 11:16:55.204011 8243 leader_election.cc:302] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783f004df8074f949239090a4e0dae47, ba7bde098c2b46e98a0b8709d2216bfc; no voters:
I20260807 11:16:55.204418 8383 raft_consensus.cc:2781] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:16:55.204536 8383 raft_consensus.cc:482] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:16:55.204658 8383 raft_consensus.cc:3037] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:55.207022 8383 raft_consensus.cc:504] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.207669 8383 leader_election.cc:288] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [CANDIDATE]: Term 1 election: Requested vote from peers ba7bde098c2b46e98a0b8709d2216bfc (127.2.223.130:46011), b54c2a3db6eb4d1593214c67f5286e50 (127.2.223.129:33365)
I20260807 11:16:55.208042 8175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d2b9f9a7f848529b6bb7f4d2e7de83" candidate_uuid: "783f004df8074f949239090a4e0dae47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7bde098c2b46e98a0b8709d2216bfc"
I20260807 11:16:55.208155 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d2b9f9a7f848529b6bb7f4d2e7de83" candidate_uuid: "783f004df8074f949239090a4e0dae47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54c2a3db6eb4d1593214c67f5286e50"
I20260807 11:16:55.208249 8175 raft_consensus.cc:3037] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:55.208391 8039 raft_consensus.cc:3037] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:55.211154 8175 raft_consensus.cc:2454] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f004df8074f949239090a4e0dae47 in term 1.
I20260807 11:16:55.211340 8039 raft_consensus.cc:2454] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f004df8074f949239090a4e0dae47 in term 1.
I20260807 11:16:55.211611 8243 leader_election.cc:302] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783f004df8074f949239090a4e0dae47, ba7bde098c2b46e98a0b8709d2216bfc; no voters:
I20260807 11:16:55.211957 8383 raft_consensus.cc:2781] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:55.212437 8383 raft_consensus.cc:686] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 1 LEADER]: Becoming Leader. State: Replica: 783f004df8074f949239090a4e0dae47, State: Running, Role: LEADER
I20260807 11:16:55.212903 8383 consensus_queue.cc:237] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } }
I20260807 11:16:55.217491 7892 catalog_manager.cc:5495] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f004df8074f949239090a4e0dae47 (127.2.223.131). New cstate: current_term: 1 leader_uuid: "783f004df8074f949239090a4e0dae47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f004df8074f949239090a4e0dae47" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 } health_report { overall_health: UNKNOWN } } }
I20260807 11:16:55.244073 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.241878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.2.223.131:42851)
0807 11:16:55.242144 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.242149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.242234 (+ 85us) server_negotiation.cc:407] Connection header received
0807 11:16:55.242413 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.242417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.242546 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.242634 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.243522 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.243525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.243528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.243553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.243578 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.243589 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.243684 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.243753 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.243798 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:16:55.252243 8039 raft_consensus.cc:1261] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 1 FOLLOWER]: Refusing update from remote peer 783f004df8074f949239090a4e0dae47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:55.252244 8175 raft_consensus.cc:1261] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 1 FOLLOWER]: Refusing update from remote peer 783f004df8074f949239090a4e0dae47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:16:55.253198 8387 consensus_queue.cc:1035] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba7bde098c2b46e98a0b8709d2216bfc" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:55.253619 8383 consensus_queue.cc:1035] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54c2a3db6eb4d1593214c67f5286e50" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:16:55.265403 8390 mvcc.cc:205] Tried to move back new op lower bound from 7315871396863066112 to 7315871396716126208. Current Snapshot: MvccSnapshot[applied={T|T < 7315871396863066112}]
I20260807 11:16:55.278767 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:55.279040 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:16:55.282792 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ada00da8566f497a9187b72cb7dda02f: Sending DeleteTablet for 3 replicas of tablet c0d2b9f9a7f848529b6bb7f4d2e7de83
I20260807 11:16:55.283918 8290 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:55 UTC) from {username='slave'} at 127.0.0.1:55282
I20260807 11:16:55.283915 8155 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:55 UTC) from {username='slave'} at 127.0.0.1:50848
I20260807 11:16:55.283968 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:16:55 UTC) from {username='slave'} at 127.0.0.1:45360
I20260807 11:16:55.286780 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7954
I20260807 11:16:55.296772 8401 tablet_replica.cc:330] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: stopping tablet replica
I20260807 11:16:55.297293 8401 raft_consensus.cc:2229] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:16:55.297763 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.288289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.2.223.129:33365)
0807 11:16:55.288476 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.288483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.293321 (+ 4838us) server_negotiation.cc:407] Connection header received
0807 11:16:55.294272 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.294281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.294423 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.294564 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.296568 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.296577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.296585 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.296613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.296665 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.296687 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.296818 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.297276 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.297362 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:16:55.299189 8081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:55.301604 8401 raft_consensus.cc:2258] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:55.309201 8401 ts_tablet_manager.cc:1905] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:55.309979 8400 tablet_replica.cc:330] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: stopping tablet replica
I20260807 11:16:55.310544 8400 raft_consensus.cc:2229] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:55.311236 8400 raft_consensus.cc:2258] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:55.313349 8400 ts_tablet_manager.cc:1905] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:16:55.320430 8400 ts_tablet_manager.cc:1918] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:55.320668 8400 log.cc:1196] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/c0d2b9f9a7f848529b6bb7f4d2e7de83
I20260807 11:16:55.321280 8400 ts_tablet_manager.cc:1939] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P ba7bde098c2b46e98a0b8709d2216bfc: Deleting consensus metadata
I20260807 11:16:55.323680 7877 catalog_manager.cc:4841] TS ba7bde098c2b46e98a0b8709d2216bfc (127.2.223.130:46011): tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 (table test-table [id=d73f63e6d5184f1d98111da8df468c6e]) successfully deleted
I20260807 11:16:55.324321 8401 ts_tablet_manager.cc:1918] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:16:55.324581 8401 log.cc:1196] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/c0d2b9f9a7f848529b6bb7f4d2e7de83
I20260807 11:16:55.325301 8401 ts_tablet_manager.cc:1939] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P 783f004df8074f949239090a4e0dae47: Deleting consensus metadata
I20260807 11:16:55.327028 7878 catalog_manager.cc:4841] TS 783f004df8074f949239090a4e0dae47 (127.2.223.131:42851): tablet c0d2b9f9a7f848529b6bb7f4d2e7de83 (table test-table [id=d73f63e6d5184f1d98111da8df468c6e]) successfully deleted
I20260807 11:16:55.449189 8399 tablet_replica.cc:330] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50: stopping tablet replica
I20260807 11:16:55.449714 8399 raft_consensus.cc:2229] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:16:55.450246 8399 raft_consensus.cc:2258] T c0d2b9f9a7f848529b6bb7f4d2e7de83 P b54c2a3db6eb4d1593214c67f5286e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:16:55.451320 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7954
W20260807 11:16:55.478881 7874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.223.129:33365 (error 108)
I20260807 11:16:55.479143 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8090
I20260807 11:16:55.481503 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.480893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45392)
0807 11:16:55.481110 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.481306 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 11:16:55.481777 7874 catalog_manager.cc:4584] TS b54c2a3db6eb4d1593214c67f5286e50 (127.2.223.129:33365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0d2b9f9a7f848529b6bb7f4d2e7de83: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
I20260807 11:16:55.483309 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.480336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.2.223.130:46011)
0807 11:16:55.480534 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.480541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.480728 (+ 187us) server_negotiation.cc:407] Connection header received
0807 11:16:55.480963 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.480969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.481082 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.481194 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.482555 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.482562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.482567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.482589 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.482642 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.482658 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.482781 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.482945 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.483009 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:16:55.484017 8218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:55.535703 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.535085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45408)
0807 11:16:55.535352 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.535466 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 11:16:55.536958 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.536524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45418)
0807 11:16:55.536706 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.536813 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 11:16:55.574064 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8090
I20260807 11:16:55.597574 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.597111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45434)
0807 11:16:55.597314 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.597400 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 11:16:55.598816 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.598362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45436)
0807 11:16:55.598583 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.598663 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260807 11:16:55.602612 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8226
I20260807 11:16:55.605916 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.603619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.2.223.131:42851)
0807 11:16:55.603809 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:55.603817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:55.603839 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:16:55.603926 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:55.603932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:55.603994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:55.604112 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:55.605165 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:55.605172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:55.605177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:55.605197 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:55.605237 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:55.605252 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:55.605395 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:55.605545 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 11:16:55.605608 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:16:55.606719 8354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:55.679450 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.678975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45446)
0807 11:16:55.679194 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.679294 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 11:16:55.680637 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.680218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45448)
0807 11:16:55.680440 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.680513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260807 11:16:55.731427 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8226
I20260807 11:16:55.756708 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 7858
I20260807 11:16:55.758023 7923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:55.867905 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:55.867236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:33365 (local address 127.0.0.1:45464)
0807 11:16:55.867493 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:55.867643 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 11:16:55.868032 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 7858
2026-08-07T11:16:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2758 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 11:16:55.914800 2942 mini_oidc.cc:152] Starting JWKS server
I20260807 11:16:55.917655 2942 webserver.cc:460] Webserver started at https://127.0.0.1:36407/ using document root <none> and password file <none>
I20260807 11:16:55.917958 2942 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:16:55.918598 2942 webserver.cc:460] Webserver started at http://127.0.0.1:35811/ using document root <none> and password file <none>
2026-08-07T11:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:55Z Disabled control of system clock
I20260807 11:16:55.953517 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35055
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:43251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35055
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/testchainca.cert with env {}
W20260807 11:16:56.283851 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:56.284243 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:56.284406 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:56.291911 8414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:56.292078 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:56.292183 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:56.292294 8414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:56.292372 8414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:16:56.292464 8414 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:16:56.292544 8414 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36407/jwks.json
I20260807 11:16:56.302973 8414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:56.304402 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:56.305784 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:56.318271 8420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:56.318389 8423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:56.318351 8421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:56.319257 8414 server_base.cc:1034] running on GCE node
I20260807 11:16:56.319977 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:56.321336 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:56.322593 8414 hybrid_clock.cc:648] HybridClock initialized: now 1786101416322544 us; error 60 us; skew 500 ppm
I20260807 11:16:56.322962 8414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:56.325352 8414 webserver.cc:460] Webserver started at http://127.2.223.190:41183/ using document root <none> and password file <none>
I20260807 11:16:56.326074 8414 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:56.326231 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:56.326633 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:56.329562 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "f88c8d1af54a4a64a83b295ad836f834"
format_stamp: "Formatted at 2026-08-07 11:16:56 on dist-test-slave-5l3k"
server_key: "42f2ba2d2f20e32661402b1f39c6bdee"
server_key_iv: "c26a8cfe4c70a2b133b21645a4c08f8c"
server_key_version: "encryptionkey@0"
I20260807 11:16:56.330382 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "f88c8d1af54a4a64a83b295ad836f834"
format_stamp: "Formatted at 2026-08-07 11:16:56 on dist-test-slave-5l3k"
server_key: "42f2ba2d2f20e32661402b1f39c6bdee"
server_key_iv: "c26a8cfe4c70a2b133b21645a4c08f8c"
server_key_version: "encryptionkey@0"
I20260807 11:16:56.335983 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:16:56.339607 8429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:56.340981 8414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:16:56.341185 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "f88c8d1af54a4a64a83b295ad836f834"
format_stamp: "Formatted at 2026-08-07 11:16:56 on dist-test-slave-5l3k"
server_key: "42f2ba2d2f20e32661402b1f39c6bdee"
server_key_iv: "c26a8cfe4c70a2b133b21645a4c08f8c"
server_key_version: "encryptionkey@0"
I20260807 11:16:56.341418 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:56.372771 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:56.373757 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:56.374122 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:56.394249 8414 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:35055
I20260807 11:16:56.394276 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:35055 every 8 connection(s)
I20260807 11:16:56.395735 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:56.399897 8484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:56.405627 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8414
I20260807 11:16:56.405853 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:56.406672 2942 external_mini_cluster.cc:1428] Setting key 68d89007050ac90c4b6a013513ec97c4
I20260807 11:16:56.412128 8484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834: Bootstrap starting.
I20260807 11:16:56.416245 8484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:56.417560 8484 log.cc:824] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:56.421016 8484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834: No bootstrap required, opened a new log
I20260807 11:16:56.421773 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:56.408772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.2.223.190:35055)
0807 11:16:56.409964 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:56.409982 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:56.410024 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:16:56.411793 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:56.411816 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:56.412305 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:56.412655 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:56.413420 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.414822 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.416955 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.417633 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.417841 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:56.419324 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:56.419332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:56.419346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:56.419888 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:56.419972 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:56.419992 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:56.420084 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:56.420357 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 11:16:56.420698 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":494,"threads_started":1}
I20260807 11:16:56.425691 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:16:56.431063 8484 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } }
I20260807 11:16:56.431411 8484 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:56.431492 8484 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88c8d1af54a4a64a83b295ad836f834, State: Initialized, Role: FOLLOWER
I20260807 11:16:56.432085 8484 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } }
I20260807 11:16:56.432307 8484 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:56.432418 8484 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:56.432559 8484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:56.435006 8484 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } }
I20260807 11:16:56.435575 8484 leader_election.cc:302] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88c8d1af54a4a64a83b295ad836f834; no voters:
I20260807 11:16:56.436178 8484 leader_election.cc:288] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:56.436282 8490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:56.437140 8490 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [term 1 LEADER]: Becoming Leader. State: Replica: f88c8d1af54a4a64a83b295ad836f834, State: Running, Role: LEADER
I20260807 11:16:56.437737 8490 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } }
I20260807 11:16:56.438707 8484 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:56.440567 8491 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88c8d1af54a4a64a83b295ad836f834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } } }
I20260807 11:16:56.440593 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88c8d1af54a4a64a83b295ad836f834. Latest consensus state: current_term: 1 leader_uuid: "f88c8d1af54a4a64a83b295ad836f834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c8d1af54a4a64a83b295ad836f834" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35055 } } }
I20260807 11:16:56.441076 8491 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:56.441205 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:56.447270 8495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:56.452509 8495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:56.462538 8495 catalog_manager.cc:1347] Generated new cluster ID: 882ebf5c0a4943dd93cda34e453366a8
I20260807 11:16:56.462677 8495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:56.474843 8495 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:56.476107 8495 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:56.485999 8495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f88c8d1af54a4a64a83b295ad836f834: Generated new TSK 0
I20260807 11:16:56.486891 8495 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 11:16:56.522688 8516 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 11:16:56.522945 8516 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 11:16:56.524179 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:56.502400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.2.223.190:35055)
0807 11:16:56.502623 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:56.502631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:56.503165 (+ 534us) server_negotiation.cc:407] Connection header received
0807 11:16:56.503606 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:56.503612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:56.503694 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:56.503855 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:16:56.504414 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.505164 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.506647 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.507083 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.507239 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:56.507817 (+ 578us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:16:56.523413 (+ 15596us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 11:16:56.523836 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 11:16:56.524329 8488 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 11:16:56.525951 8515 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:16:56.502305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35055 (local address 127.0.0.1:52876)
0807 11:16:56.503046 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:56.503063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:56.503283 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:56.503888 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:56.503893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:56.503906 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:16:56.504257 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:56.504266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.505323 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.505335 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:56.506443 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:56.506454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:56.507269 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:56.507275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:56.507441 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:56.507499 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:16:56.523777 (+ 16278us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 11:16:56.524143 (+ 366us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.223.190:35055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":424,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.223.190:35055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 11:16:56.527722 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8414
I20260807 11:16:56.528923 8478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:56.623773 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8414
2026-08-07T11:16:56Z chronyd exiting
I20260807 11:16:56.679179 2942 test_util.cc:175] -----------------------------------------------
I20260807 11:16:56.679329 2942 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786101369622807-2942-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (769 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 11:16:56.683374 2942 mini_oidc.cc:152] Starting JWKS server
I20260807 11:16:56.685276 2942 webserver.cc:460] Webserver started at https://127.0.0.1:36055/ using document root <none> and password file <none>
I20260807 11:16:56.685521 2942 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:16:56.686236 2942 webserver.cc:460] Webserver started at http://127.0.0.1:43377/ using document root <none> and password file <none>
2026-08-07T11:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:56Z Disabled control of system clock
I20260807 11:16:56.724990 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:44193
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:44193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36055/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/testchainca.cert with env {}
W20260807 11:16:57.070461 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:57.070883 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:57.071063 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:57.078215 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:57.078359 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:57.078454 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:57.078588 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:57.078676 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:16:57.078768 8523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:16:57.078851 8523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36055/jwks.json
I20260807 11:16:57.088939 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:44193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36055/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:57.090253 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:57.091575 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:57.103864 8529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:57.103968 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:57.103901 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:57.105088 8523 server_base.cc:1034] running on GCE node
I20260807 11:16:57.105804 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:57.107189 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:57.108392 8523 hybrid_clock.cc:648] HybridClock initialized: now 1786101417108335 us; error 53 us; skew 500 ppm
I20260807 11:16:57.108703 8523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:57.119168 8523 webserver.cc:460] Webserver started at http://127.2.223.190:35285/ using document root <none> and password file <none>
I20260807 11:16:57.119897 8523 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:57.119994 8523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:57.120292 8523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:57.123333 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "2747d84693978636723f0d5341a58da7"
server_key_iv: "dd434b00a165421da1752be3aa836c89"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.124079 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "2747d84693978636723f0d5341a58da7"
server_key_iv: "dd434b00a165421da1752be3aa836c89"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.129254 8523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:16:57.132623 8537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:57.133896 8523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 11:16:57.134086 8523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "2747d84693978636723f0d5341a58da7"
server_key_iv: "dd434b00a165421da1752be3aa836c89"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.134299 8523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:57.149325 8523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:57.150096 8523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:57.150401 8523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:57.168972 8523 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:44193
I20260807 11:16:57.169054 8592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:44193 every 8 connection(s)
I20260807 11:16:57.170305 8523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:57.173918 8593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:57.179050 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8523
I20260807 11:16:57.179246 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:57.179893 2942 external_mini_cluster.cc:1428] Setting key 0d6df26cb9bdac1c581527796b8fa78d
I20260807 11:16:57.184589 8593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf: Bootstrap starting.
I20260807 11:16:57.188066 8593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:57.189275 8593 log.cc:824] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:57.192000 8593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf: No bootstrap required, opened a new log
I20260807 11:16:57.193341 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:57.181798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.2.223.190:44193)
0807 11:16:57.182894 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:57.182905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:57.182947 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:16:57.184614 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:57.184637 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:57.185112 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:57.185407 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:57.186030 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.187347 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.189145 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.189833 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.190043 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:57.191194 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:57.191201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:57.191213 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:57.191743 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:57.191815 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:57.191835 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:57.191960 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:57.192191 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 11:16:57.192481 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
I20260807 11:16:57.196985 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:16:57.200204 8593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } }
I20260807 11:16:57.200623 8593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:57.200736 8593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf, State: Initialized, Role: FOLLOWER
I20260807 11:16:57.201274 8593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } }
I20260807 11:16:57.201531 8593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:57.201687 8593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:57.201915 8593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:57.204268 8593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } }
I20260807 11:16:57.204767 8593 leader_election.cc:302] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf; no voters:
I20260807 11:16:57.205349 8593 leader_election.cc:288] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:57.205502 8598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:57.206810 8598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [term 1 LEADER]: Becoming Leader. State: Replica: 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf, State: Running, Role: LEADER
I20260807 11:16:57.207306 8598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } }
I20260807 11:16:57.207638 8593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:57.210439 8599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } } }
I20260807 11:16:57.210933 8599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:57.210389 8600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf. Latest consensus state: current_term: 1 leader_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a186f5b6c2b4ffb8d2c4d6ec68dbbcf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 44193 } } }
I20260807 11:16:57.211576 8600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:57.212006 8607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:57.218365 8607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:57.227005 8607 catalog_manager.cc:1347] Generated new cluster ID: f7c1dee110fd4f24bd8009c109bf34ce
I20260807 11:16:57.227139 8607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:57.256402 8607 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:57.258328 8607 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:57.267556 8607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a186f5b6c2b4ffb8d2c4d6ec68dbbcf: Generated new TSK 0
I20260807 11:16:57.268594 8607 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 11:16:57.308748 8623 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 11:16:57.308874 8623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260807 11:16:57.310137 8622 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:16:57.288919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:44193 (local address 127.0.0.1:53392)
0807 11:16:57.289692 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:57.289708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:57.289834 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:57.290373 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:57.290376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:57.290385 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:16:57.290700 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:57.290709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.292181 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.292185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:57.293273 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:57.293283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.294304 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.294308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:57.294450 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:57.294550 (+ 100us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:16:57.309648 (+ 15098us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 11:16:57.309855 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.223.190:44193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":449,"threads_started":1}
I20260807 11:16:57.310128 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:57.289094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.2.223.190:44193)
0807 11:16:57.289300 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:57.289307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:57.289796 (+ 489us) server_negotiation.cc:407] Connection header received
0807 11:16:57.290036 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:57.290042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:57.290163 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:57.290344 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:16:57.290872 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.292033 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.293416 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.294120 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.294255 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:57.294840 (+ 585us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:16:57.309402 (+ 14562us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 11:16:57.309796 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 11:16:57.310276 8596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260807 11:16:57.311589 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8523
I20260807 11:16:57.312912 8588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:57.408864 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8523
2026-08-07T11:16:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (787 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 11:16:57.470323 2942 mini_oidc.cc:152] Starting JWKS server
I20260807 11:16:57.472313 2942 webserver.cc:460] Webserver started at https://127.0.0.1:42453/ using document root <none> and password file <none>
I20260807 11:16:57.472584 2942 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:16:57.473290 2942 webserver.cc:460] Webserver started at http://127.0.0.1:41811/ using document root <none> and password file <none>
2026-08-07T11:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:57Z Disabled control of system clock
I20260807 11:16:57.511062 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33355
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33355
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42453/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 11:16:57.846246 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:57.846612 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:57.846791 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:57.854136 8631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:57.854277 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:57.854372 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:57.854521 8631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:57.854607 8631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:16:57.854701 8631 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:16:57.854781 8631 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42453/jwks.json
I20260807 11:16:57.864305 8631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42453/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:57.865677 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:57.867642 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:57.879937 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:57.879999 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:57.880546 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:57.880926 8631 server_base.cc:1034] running on GCE node
I20260807 11:16:57.881757 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:57.883239 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:57.884440 8631 hybrid_clock.cc:648] HybridClock initialized: now 1786101417884392 us; error 43 us; skew 500 ppm
I20260807 11:16:57.884791 8631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:57.887091 8631 webserver.cc:460] Webserver started at http://127.2.223.190:32891/ using document root <none> and password file <none>
I20260807 11:16:57.887775 8631 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:57.887888 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:57.888226 8631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:57.891312 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "032c5970d7334cf7873691b3ef86b5ef"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "6dc5c32a7df74ee3419c42b29639548d"
server_key_iv: "3e7565dd5763efec86cfd2aae28e96ea"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.892139 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "032c5970d7334cf7873691b3ef86b5ef"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "6dc5c32a7df74ee3419c42b29639548d"
server_key_iv: "3e7565dd5763efec86cfd2aae28e96ea"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.897208 8631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:16:57.900614 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:57.901938 8631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:16:57.902151 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "032c5970d7334cf7873691b3ef86b5ef"
format_stamp: "Formatted at 2026-08-07 11:16:57 on dist-test-slave-5l3k"
server_key: "6dc5c32a7df74ee3419c42b29639548d"
server_key_iv: "3e7565dd5763efec86cfd2aae28e96ea"
server_key_version: "encryptionkey@0"
I20260807 11:16:57.902395 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:57.926543 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:57.927453 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:57.927790 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:57.946779 8631 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33355
I20260807 11:16:57.946827 8700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33355 every 8 connection(s)
I20260807 11:16:57.948148 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:57.952240 8701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:57.954335 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8631
I20260807 11:16:57.954612 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:57.955430 2942 external_mini_cluster.cc:1428] Setting key 47efe90057dd64c96bb66898bc137ea7
I20260807 11:16:57.964757 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef: Bootstrap starting.
I20260807 11:16:57.967895 8701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:57.969035 8701 log.cc:824] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:57.969142 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:57.957480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.2.223.190:33355)
0807 11:16:57.958658 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:57.958670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:57.958714 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:16:57.960401 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:57.960424 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:57.960872 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:57.961268 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:57.961855 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.963269 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.964607 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:57.965348 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:57.965564 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:57.966834 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:57.966842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:57.966855 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:57.967393 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:57.967484 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:57.967506 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:57.967637 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:57.967849 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 11:16:57.968177 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":443,"threads_started":1}
I20260807 11:16:57.972132 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef: No bootstrap required, opened a new log
I20260807 11:16:57.973004 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:16:57.980294 8701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } }
I20260807 11:16:57.980690 8701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:57.980801 8701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032c5970d7334cf7873691b3ef86b5ef, State: Initialized, Role: FOLLOWER
I20260807 11:16:57.981474 8701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } }
I20260807 11:16:57.981729 8701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:57.981892 8701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:57.982067 8701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:57.984697 8701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } }
I20260807 11:16:57.985235 8701 leader_election.cc:302] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032c5970d7334cf7873691b3ef86b5ef; no voters:
I20260807 11:16:57.985852 8701 leader_election.cc:288] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:57.986001 8707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:57.986760 8707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [term 1 LEADER]: Becoming Leader. State: Replica: 032c5970d7334cf7873691b3ef86b5ef, State: Running, Role: LEADER
I20260807 11:16:57.987412 8707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } }
I20260807 11:16:57.988020 8701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:57.990302 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032c5970d7334cf7873691b3ef86b5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } } }
I20260807 11:16:57.990705 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:57.990705 8710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032c5970d7334cf7873691b3ef86b5ef. Latest consensus state: current_term: 1 leader_uuid: "032c5970d7334cf7873691b3ef86b5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c5970d7334cf7873691b3ef86b5ef" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33355 } } }
I20260807 11:16:57.991549 8710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:57.992712 8714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:57.998175 8714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:58.007439 8714 catalog_manager.cc:1347] Generated new cluster ID: 83921866399a4d2696a2a7e9e4dcf420
I20260807 11:16:58.007548 8714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:58.028765 8714 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:58.030035 8714 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:58.037624 8714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032c5970d7334cf7873691b3ef86b5ef: Generated new TSK 0
I20260807 11:16:58.038393 8714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:16:58.069630 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:58.060959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.2.223.190:33355)
0807 11:16:58.061175 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:58.061184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:58.061660 (+ 476us) server_negotiation.cc:407] Connection header received
0807 11:16:58.061872 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:58.061877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:58.061972 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:58.062180 (+ 208us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:16:58.062683 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.063814 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.065226 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.065949 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.066075 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:58.066601 (+ 526us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:16:58.068981 (+ 2380us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 11:16:58.069320 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:16:58.069765 8704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 11:16:58.071439 8733 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:16:58.060844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33355 (local address 127.0.0.1:40674)
0807 11:16:58.061556 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:58.061572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:58.061695 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:58.062186 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:58.062190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:58.062200 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:16:58.062453 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:58.062461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.063987 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.063991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:58.065075 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:58.065089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.066153 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.066158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:58.066288 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:58.066320 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:16:58.069211 (+ 2891us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 11:16:58.071046 (+ 1835us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.223.190:33355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":1505,"thread_start_us":418,"threads_started":1}
I20260807 11:16:58.074525 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8631
I20260807 11:16:58.075696 8696 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 11:16:58.081846 8734 webserver.cc:533] Webserver: sslize failed:
E20260807 11:16:58.081964 8734 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 11:16:58.170547 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8631
2026-08-07T11:16:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T11:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:16:58Z Disabled control of system clock
I20260807 11:16:58.264925 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41245
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41245
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/testchainca.cert with env {}
W20260807 11:16:58.598804 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:16:58.599144 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:16:58.599236 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:16:58.606712 8739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:16:58.606829 8739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:16:58.606868 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:16:58.606931 8739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:16:58.606976 8739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:16:58.607041 8739 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:16:58.607082 8739 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 11:16:58.617419 8739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:16:58.618808 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:16:58.620111 8739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:16:58.632275 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:58.632251 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:16:58.632495 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:16:58.632992 8739 server_base.cc:1034] running on GCE node
I20260807 11:16:58.633842 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:16:58.635484 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:16:58.636730 8739 hybrid_clock.cc:648] HybridClock initialized: now 1786101418636690 us; error 40 us; skew 500 ppm
I20260807 11:16:58.637118 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:16:58.639528 8739 webserver.cc:460] Webserver started at http://127.2.223.190:42397/ using document root <none> and password file <none>
I20260807 11:16:58.640249 8739 fs_manager.cc:359] Metadata directory not provided
I20260807 11:16:58.640390 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:16:58.640730 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:16:58.643765 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "d26bc3f449c0441eaab93f122e475cf6"
format_stamp: "Formatted at 2026-08-07 11:16:58 on dist-test-slave-5l3k"
server_key: "79e8e38777a5a8694087406af67975b6"
server_key_iv: "3c75d96110449718feac0fb91d2627a8"
server_key_version: "encryptionkey@0"
I20260807 11:16:58.644622 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "d26bc3f449c0441eaab93f122e475cf6"
format_stamp: "Formatted at 2026-08-07 11:16:58 on dist-test-slave-5l3k"
server_key: "79e8e38777a5a8694087406af67975b6"
server_key_iv: "3c75d96110449718feac0fb91d2627a8"
server_key_version: "encryptionkey@0"
I20260807 11:16:58.650147 8739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:16:58.653940 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:16:58.655725 8739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:16:58.655963 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "d26bc3f449c0441eaab93f122e475cf6"
format_stamp: "Formatted at 2026-08-07 11:16:58 on dist-test-slave-5l3k"
server_key: "79e8e38777a5a8694087406af67975b6"
server_key_iv: "3c75d96110449718feac0fb91d2627a8"
server_key_version: "encryptionkey@0"
I20260807 11:16:58.656245 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:16:58.678619 8739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:16:58.679577 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:16:58.679899 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:16:58.699717 8739 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41245
I20260807 11:16:58.699750 8808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41245 every 8 connection(s)
I20260807 11:16:58.701150 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:16:58.705090 8809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:16:58.707065 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8739
I20260807 11:16:58.707306 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:16:58.707963 2942 external_mini_cluster.cc:1428] Setting key 53c2c9ad5d8f82436aad6a40dc535f9c
I20260807 11:16:58.719012 8809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6: Bootstrap starting.
I20260807 11:16:58.723217 8809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6: Neither blocks nor log segments found. Creating new log.
I20260807 11:16:58.724458 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:58.710028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.2.223.190:41245)
0807 11:16:58.711179 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:58.711195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:58.711248 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:16:58.713652 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:58.713684 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:58.714300 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:58.714776 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:16:58.715352 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.717440 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.718858 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.719903 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.720093 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:58.721407 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:16:58.721416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:16:58.721437 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:16:58.722163 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:16:58.722288 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:16:58.722317 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:16:58.722434 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 11:16:58.722708 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 11:16:58.723088 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":397,"threads_started":1}
I20260807 11:16:58.724607 8809 log.cc:824] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:16:58.727950 8809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6: No bootstrap required, opened a new log
I20260807 11:16:58.728783 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:16:58.735802 8809 raft_consensus.cc:348] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } }
I20260807 11:16:58.736275 8809 raft_consensus.cc:374] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:16:58.736418 8809 raft_consensus.cc:729] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26bc3f449c0441eaab93f122e475cf6, State: Initialized, Role: FOLLOWER
I20260807 11:16:58.737004 8809 consensus_queue.cc:260] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } }
I20260807 11:16:58.737237 8809 raft_consensus.cc:388] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:16:58.737397 8809 raft_consensus.cc:482] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:16:58.737586 8809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:16:58.739920 8809 raft_consensus.cc:504] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } }
I20260807 11:16:58.740484 8809 leader_election.cc:302] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d26bc3f449c0441eaab93f122e475cf6; no voters:
I20260807 11:16:58.741135 8809 leader_election.cc:288] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:16:58.741374 8814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:16:58.742731 8814 raft_consensus.cc:686] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [term 1 LEADER]: Becoming Leader. State: Replica: d26bc3f449c0441eaab93f122e475cf6, State: Running, Role: LEADER
I20260807 11:16:58.743256 8814 consensus_queue.cc:237] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } }
I20260807 11:16:58.743669 8809 sys_catalog.cc:564] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:16:58.746640 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d26bc3f449c0441eaab93f122e475cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } } }
I20260807 11:16:58.747011 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:58.747483 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d26bc3f449c0441eaab93f122e475cf6. Latest consensus state: current_term: 1 leader_uuid: "d26bc3f449c0441eaab93f122e475cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26bc3f449c0441eaab93f122e475cf6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41245 } } }
I20260807 11:16:58.747817 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:16:58.750118 8820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:16:58.755393 8820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:16:58.764492 8820 catalog_manager.cc:1347] Generated new cluster ID: 53686de76b3c4236a17689d4f5fe1f66
I20260807 11:16:58.764642 8820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:16:58.787814 8820 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:16:58.789778 8820 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:16:58.798681 8820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d26bc3f449c0441eaab93f122e475cf6: Generated new TSK 0
I20260807 11:16:58.799800 8820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:16:58.849772 8839 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:16:58.815217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41245 (local address 127.0.0.1:52882)
0807 11:16:58.815802 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 11:16:58.815823 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:16:58.816003 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:16:58.816484 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:16:58.816490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:16:58.816501 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:16:58.816812 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:58.816821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.817892 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.817897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:58.819090 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:16:58.819102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.819863 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.819870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:16:58.820025 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:58.820058 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:16:58.848885 (+ 28827us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 11:16:58.849224 (+ 339us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.223.190:41245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":390,"threads_started":1}
I20260807 11:16:58.850824 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:16:58.815369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.2.223.190:41245)
0807 11:16:58.815589 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:16:58.815596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:16:58.815920 (+ 324us) server_negotiation.cc:407] Connection header received
0807 11:16:58.816216 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:16:58.816220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:16:58.816297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:16:58.816475 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:16:58.816966 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.817745 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.819232 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:16:58.819682 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:16:58.819824 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:16:58.820316 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:16:58.848473 (+ 28157us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 11:16:58.850414 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52882: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 11:16:58.851553 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8739
W20260807 11:16:58.851584 8812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52882: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260807 11:16:58.852936 8804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:16:58.954622 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8739
2026-08-07T11:16:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (763 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (426 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (433 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:16:59 dist-test-slave-5l3k krb5kdc[8856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:16:59 dist-test-slave-5l3k krb5kdc[8856](info): set up 2 sockets
Aug 07 11:16:59 dist-test-slave-5l3k krb5kdc[8856](info): commencing operation
krb5kdc: starting...
W20260807 11:17:00.972195 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:01 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:17:01.222785 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33571
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:01.558328 8869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:01.558703 8869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:01.558888 8869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:01.566232 8869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:01.566336 8869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:01.566448 8869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:01.566603 8869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:01.566687 8869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:01.576617 8869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:01.577998 8869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:01.579381 8869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:01.591847 8874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:01.591840 8875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:01.591907 8877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:01.592903 8869 server_base.cc:1034] running on GCE node
I20260807 11:17:01.593734 8869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:01.595284 8869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:01.596520 8869 hybrid_clock.cc:648] HybridClock initialized: now 1786101421596477 us; error 32 us; skew 500 ppm
I20260807 11:17:01.596928 8869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:01 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101421, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:01.600234 8869 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:17:01.601660 8869 webserver.cc:460] Webserver started at http://127.2.223.190:35703/ using document root <none> and password file <none>
I20260807 11:17:01.602344 8869 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:01.602541 8869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:01.602880 8869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:01.605834 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "f33f05d425b0477fb7960fea6e5ca8c5"
format_stamp: "Formatted at 2026-08-07 11:17:01 on dist-test-slave-5l3k"
server_key: "ca7db00ea5de341a37d11711ec8f575c"
server_key_iv: "834fb334c079bb3765c08ea89660f984"
server_key_version: "encryptionkey@0"
I20260807 11:17:01.606702 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "f33f05d425b0477fb7960fea6e5ca8c5"
format_stamp: "Formatted at 2026-08-07 11:17:01 on dist-test-slave-5l3k"
server_key: "ca7db00ea5de341a37d11711ec8f575c"
server_key_iv: "834fb334c079bb3765c08ea89660f984"
server_key_version: "encryptionkey@0"
I20260807 11:17:01.611876 8869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:17:01.615257 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:01.616519 8869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:01.616706 8869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "f33f05d425b0477fb7960fea6e5ca8c5"
format_stamp: "Formatted at 2026-08-07 11:17:01 on dist-test-slave-5l3k"
server_key: "ca7db00ea5de341a37d11711ec8f575c"
server_key_iv: "834fb334c079bb3765c08ea89660f984"
server_key_version: "encryptionkey@0"
I20260807 11:17:01.616900 8869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:01.632654 8869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:01.637015 8869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:01.637295 8869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:01.656001 8869 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33571
I20260807 11:17:01.656040 8943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33571 every 8 connection(s)
I20260807 11:17:01.657397 8869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:01.657850 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8869
I20260807 11:17:01.658092 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:01.658982 2942 external_mini_cluster.cc:1428] Setting key e0579a248ff41e301dfb3d3bc6a57d76
I20260807 11:17:01.661793 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:17:01 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:01.675542 8944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5: Bootstrap starting.
I20260807 11:17:01.678772 8944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:01.679804 8944 log.cc:824] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:01.683523 8944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5: No bootstrap required, opened a new log
I20260807 11:17:01.683650 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:01.661259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.2.223.190:33571)
0807 11:17:01.662124 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:01.662137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:01.662180 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:17:01.664184 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:01.664206 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:01.664579 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:01.664907 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:01.666754 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:01.668195 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:01.670026 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:01.670809 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:01.670944 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:01.675689 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:01.675699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:01.675718 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:01.675774 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:01.678971 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:01.679686 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:01.679701 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:01.679708 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:01.679892 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:01.680384 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:01.680392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:01.680397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:01.681156 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:01.681423 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:01.681918 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 11:17:01.682274 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":376,"threads_started":1}
I20260807 11:17:01.691836 8944 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } }
I20260807 11:17:01.692159 8944 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:01.692239 8944 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f05d425b0477fb7960fea6e5ca8c5, State: Initialized, Role: FOLLOWER
I20260807 11:17:01.692795 8944 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } }
I20260807 11:17:01.693058 8944 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:01.693219 8944 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:01.693423 8944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:01.695595 8944 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } }
I20260807 11:17:01.696059 8944 leader_election.cc:302] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33f05d425b0477fb7960fea6e5ca8c5; no voters:
I20260807 11:17:01.696651 8944 leader_election.cc:288] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:01.696789 8949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:01.697530 8949 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [term 1 LEADER]: Becoming Leader. State: Replica: f33f05d425b0477fb7960fea6e5ca8c5, State: Running, Role: LEADER
I20260807 11:17:01.698071 8949 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } }
I20260807 11:17:01.698845 8944 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:01.701277 8951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33f05d425b0477fb7960fea6e5ca8c5. Latest consensus state: current_term: 1 leader_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } } }
I20260807 11:17:01.701695 8951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:01.701061 8950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33f05d425b0477fb7960fea6e5ca8c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33571 } } }
I20260807 11:17:01.702092 8950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:01.702893 8957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:01.710187 8957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:01.718554 8957 catalog_manager.cc:1347] Generated new cluster ID: 1934782eb3f9499db0f4970dadc8b26e
I20260807 11:17:01.718676 8957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:01.738144 8957 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:01.739526 8957 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:01.747244 8957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33f05d425b0477fb7960fea6e5ca8c5: Generated new TSK 0
I20260807 11:17:01.748061 8957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:17:01.809749 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:02.184284 8970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:02.184661 8970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:02.184772 8970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:02.191902 8970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:02.192009 8970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:02.192118 8970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:02.201685 8970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:02.202907 8970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:02.204254 8970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:02.218315 8975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:02.218386 8976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:02.218359 8978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:02.219314 8970 server_base.cc:1034] running on GCE node
I20260807 11:17:02.219892 8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:02.220928 8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:02.222163 8970 hybrid_clock.cc:648] HybridClock initialized: now 1786101422222079 us; error 91 us; skew 500 ppm
I20260807 11:17:02.222608 8970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:02 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:02.229323 8970 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:17:02.230966 8970 webserver.cc:460] Webserver started at http://127.2.223.129:36273/ using document root <none> and password file <none>
I20260807 11:17:02.231680 8970 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:02.231835 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:02.232167 8970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:02.235230 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "fd1d263035d54410b153f65b7b12bc7d"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "580d699503759c3ae77d74d5a904a5aa"
server_key_iv: "edf632bb2e3b4068df9f6f24f0e86c51"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.236088 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "fd1d263035d54410b153f65b7b12bc7d"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "580d699503759c3ae77d74d5a904a5aa"
server_key_iv: "edf632bb2e3b4068df9f6f24f0e86c51"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.241389 8970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:17:02.244812 8987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.246107 8970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:02.246279 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "fd1d263035d54410b153f65b7b12bc7d"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "580d699503759c3ae77d74d5a904a5aa"
server_key_iv: "edf632bb2e3b4068df9f6f24f0e86c51"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.246546 8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:02.264813 8970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:02.268978 8970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:02.269295 8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:02.270280 8970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:02.271664 8970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:02.271773 8970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.271938 8970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:02.272013 8970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.306041 8970 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:34847
I20260807 11:17:02.306115 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:34847 every 8 connection(s)
I20260807 11:17:02.307436 8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:02.317077 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 8970
I20260807 11:17:02.317301 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:02.318004 2942 external_mini_cluster.cc:1428] Setting key 722743bf295fb610cd575eff832e8f80
Aug 07 11:17:02 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:02.327768 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:02.309875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:35563 (local address 127.2.223.190:33571)
0807 11:17:02.310362 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:02.310385 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:02.311469 (+ 1084us) server_negotiation.cc:407] Connection header received
0807 11:17:02.313266 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:02.313271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:02.313336 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:02.313430 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:02.314898 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.315631 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.317481 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.317889 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.317989 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:02.321938 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:02.321943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:02.321946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:02.321976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:02.324695 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:02.325462 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:02.325472 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:02.325475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:02.325600 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:02.326069 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:02.326073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:02.326075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:02.326328 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:02.326548 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:02.327213 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 11:17:02.327424 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260807 11:17:02.328711 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:02.310533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33571 (local address 127.2.223.129:35563)
0807 11:17:02.311311 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:02.311348 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:02.313014 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:02.313643 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:02.313650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:02.314172 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:02.314756 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:02.314772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.315765 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.315768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:02.317334 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:02.317351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.318147 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.318156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:02.318429 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:02.319253 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:02.319280 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:02.321730 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:02.324851 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:02.324859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:02.324873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:02.325323 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:02.325710 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:02.325733 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:02.325736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:02.325919 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:02.326456 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:02.326463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:02.326836 (+ 373us) client_negotiation.cc:769] Sending connection context
0807 11:17:02.327087 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 11:17:02.327577 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":301,"threads_started":1}
I20260807 11:17:02.330279 9103 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33571
I20260807 11:17:02.330641 9103 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:02.331382 9103 heartbeater.cc:511] Master 127.2.223.190:33571 requested a full tablet report, sending...
I20260807 11:17:02.333560 8902 ts_manager.cc:194] Registered new tserver with Master: fd1d263035d54410b153f65b7b12bc7d (127.2.223.129:34847)
I20260807 11:17:02.335094 8902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:35563
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:17:02.375010 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:02.721406 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:02.721694 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:02.721789 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:02.728972 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:02.729079 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:02.729190 9109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:02.739147 9109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:02.740357 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:02.741612 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:02.755482 9114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:02.755532 9117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:02.755578 9115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:02.757037 9109 server_base.cc:1034] running on GCE node
I20260807 11:17:02.757689 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:02.758719 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:02.759936 9109 hybrid_clock.cc:648] HybridClock initialized: now 1786101422759911 us; error 96 us; skew 500 ppm
I20260807 11:17:02.760245 9109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:02 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:02.763387 9109 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:17:02.764752 9109 webserver.cc:460] Webserver started at http://127.2.223.130:45969/ using document root <none> and password file <none>
I20260807 11:17:02.765456 9109 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:02.765607 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:02.765921 9109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:02.769016 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "ed5722805bcf426e9582e95088e93072"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "ff168d86b01c602cde2a3aeb4cf5cb73"
server_key_iv: "7f4519b953e4babcee129103bedb3b2e"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.769832 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "ed5722805bcf426e9582e95088e93072"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "ff168d86b01c602cde2a3aeb4cf5cb73"
server_key_iv: "7f4519b953e4babcee129103bedb3b2e"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.775218 9109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:17:02.778733 9124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.780062 9109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:17:02.780256 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "ed5722805bcf426e9582e95088e93072"
format_stamp: "Formatted at 2026-08-07 11:17:02 on dist-test-slave-5l3k"
server_key: "ff168d86b01c602cde2a3aeb4cf5cb73"
server_key_iv: "7f4519b953e4babcee129103bedb3b2e"
server_key_version: "encryptionkey@0"
I20260807 11:17:02.780467 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:02.798139 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:02.802387 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:02.802755 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:02.803772 9109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:02.805224 9109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:02.805330 9109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.805495 9109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:02.805570 9109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:02.840996 9109 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:34929
I20260807 11:17:02.841037 9245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:34929 every 8 connection(s)
I20260807 11:17:02.842263 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:02.850481 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9109
I20260807 11:17:02.850699 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:02.851372 2942 external_mini_cluster.cc:1428] Setting key d53ca7ac9a364a06f40010c166dfe159
Aug 07 11:17:02 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:02.864965 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:02.844890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:56035 (local address 127.2.223.190:33571)
0807 11:17:02.845360 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:02.845364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:02.846238 (+ 874us) server_negotiation.cc:407] Connection header received
0807 11:17:02.848142 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:02.848147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:02.848250 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:02.848409 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:02.850736 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.852034 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.854373 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.854886 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.855051 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:02.858888 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:02.858893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:02.858896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:02.858931 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:02.861735 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:02.862558 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:02.862565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:02.862569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:02.862710 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:02.863187 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:02.863199 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:02.863201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:02.863496 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:02.863667 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:02.864503 (+ 836us) server_negotiation.cc:299] Negotiation successful
0807 11:17:02.864691 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260807 11:17:02.866626 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:02.845346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33571 (local address 127.2.223.130:56035)
0807 11:17:02.846077 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:02.846116 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:02.847789 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:02.848683 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:02.848698 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:02.849572 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:02.850464 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:02.850523 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.852260 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.852270 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:02.854214 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:02.854234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:02.855171 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:02.855181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:02.855500 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:02.856204 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:02.856232 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:02.858640 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:02.861905 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:02.861915 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:02.861934 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:02.862396 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:02.862840 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:02.862846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:02.862849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:02.863060 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:02.863638 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:02.863647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:02.864017 (+ 370us) client_negotiation.cc:769] Sending connection context
0807 11:17:02.864388 (+ 371us) client_negotiation.cc:240] Negotiation successful
0807 11:17:02.865046 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":298,"threads_started":1}
I20260807 11:17:02.868157 9246 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33571
I20260807 11:17:02.868484 9246 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:02.869246 9246 heartbeater.cc:511] Master 127.2.223.190:33571 requested a full tablet report, sending...
I20260807 11:17:02.870796 8902 ts_manager.cc:194] Registered new tserver with Master: ed5722805bcf426e9582e95088e93072 (127.2.223.130:34929)
I20260807 11:17:02.871639 8902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:56035
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:17:02.910279 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:03.249539 9252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:03.249800 9252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:03.249895 9252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:03.257164 9252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:03.257272 9252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:03.257388 9252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:03.267086 9252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:03.268425 9252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:03.269731 9252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:03.283471 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:03.283546 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:03.283466 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:03.284631 9252 server_base.cc:1034] running on GCE node
I20260807 11:17:03.285320 9252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:03.286279 9252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:03.287516 9252 hybrid_clock.cc:648] HybridClock initialized: now 1786101423287462 us; error 43 us; skew 500 ppm
I20260807 11:17:03.287874 9252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.291137 9252 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:17:03.292573 9252 webserver.cc:460] Webserver started at http://127.2.223.131:41531/ using document root <none> and password file <none>
I20260807 11:17:03.293270 9252 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:03.293422 9252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:03.293747 9252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:03.296726 9252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "22dcc34401cb423281a5d8f2a29b4375"
format_stamp: "Formatted at 2026-08-07 11:17:03 on dist-test-slave-5l3k"
server_key: "77b3d942dc3725ed20f6800870e685a2"
server_key_iv: "1a9d17d931087abbcd161c055c9eb396"
server_key_version: "encryptionkey@0"
I20260807 11:17:03.297542 9252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "22dcc34401cb423281a5d8f2a29b4375"
format_stamp: "Formatted at 2026-08-07 11:17:03 on dist-test-slave-5l3k"
server_key: "77b3d942dc3725ed20f6800870e685a2"
server_key_iv: "1a9d17d931087abbcd161c055c9eb396"
server_key_version: "encryptionkey@0"
I20260807 11:17:03.302821 9252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 11:17:03.306603 9269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:03.308145 9252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:03.308295 9252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "22dcc34401cb423281a5d8f2a29b4375"
format_stamp: "Formatted at 2026-08-07 11:17:03 on dist-test-slave-5l3k"
server_key: "77b3d942dc3725ed20f6800870e685a2"
server_key_iv: "1a9d17d931087abbcd161c055c9eb396"
server_key_version: "encryptionkey@0"
I20260807 11:17:03.308513 9252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:03.326560 9252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:03.330379 9252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:03.330706 9252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:03.331595 9252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:03.332907 9252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:03.332980 9252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:03.333070 9252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:03.333132 9252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:03.337971 9103 heartbeater.cc:503] Master 127.2.223.190:33571 was elected leader, sending a full tablet report...
I20260807 11:17:03.369045 9252 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:44585
I20260807 11:17:03.369140 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:44585 every 8 connection(s)
I20260807 11:17:03.370337 9252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:03.375640 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9252
I20260807 11:17:03.375830 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:03.376274 2942 external_mini_cluster.cc:1428] Setting key 5d99f368f61d0fc70adcaa225accaf88
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:03.391686 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.373183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:51971 (local address 127.2.223.190:33571)
0807 11:17:03.373679 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.373683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.374708 (+ 1025us) server_negotiation.cc:407] Connection header received
0807 11:17:03.376660 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.376666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.376759 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.376883 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.378957 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.380300 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.381882 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.382266 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.382361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.385947 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.385953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.385956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.385989 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.388680 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.389383 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.389386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.389389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.389503 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.389865 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.389868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.389870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.390123 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.390238 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.391094 (+ 856us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.391293 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":231,"threads_started":1}
I20260807 11:17:03.392807 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.373727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33571 (local address 127.2.223.131:51971)
0807 11:17:03.374533 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:03.374595 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:03.376344 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:03.377213 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:03.377226 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:03.378026 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:03.378786 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:03.378806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.380455 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.380459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:03.381751 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:03.381764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.382462 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.382467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:03.382748 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.383355 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:03.383382 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:03.385719 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:03.388823 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:03.388830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:03.388843 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:03.389273 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:03.389595 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:03.389598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:03.389600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:03.389767 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:03.390250 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:03.390257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:03.390693 (+ 436us) client_negotiation.cc:769] Sending connection context
0807 11:17:03.390952 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 11:17:03.391517 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":318,"threads_started":1}
I20260807 11:17:03.394197 9389 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33571
I20260807 11:17:03.394585 9389 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:03.395339 9389 heartbeater.cc:511] Master 127.2.223.190:33571 requested a full tablet report, sending...
I20260807 11:17:03.396692 8902 ts_manager.cc:194] Registered new tserver with Master: 22dcc34401cb423281a5d8f2a29b4375 (127.2.223.131:44585)
I20260807 11:17:03.397469 8902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:51971
I20260807 11:17:03.404670 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/test-admin.keytab.
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.440369 2942 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 11:17:03.445904 9400 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:03.443080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.2.223.129:34847)
0807 11:17:03.443746 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.443766 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.443802 (+ 36us) server_negotiation.cc:407] Connection header received
0807 11:17:03.444001 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.444009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.444235 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.444408 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.445453 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":294,"threads_started":1}
W20260807 11:17:03.446420 2942 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.448201 2942 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.449568 2942 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:17:03.465090 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.451355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.2.223.129:34847)
0807 11:17:03.451542 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.451546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.451712 (+ 166us) server_negotiation.cc:407] Connection header received
0807 11:17:03.451803 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.451806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.451864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.451952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.452982 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.453943 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.455222 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.455787 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.455876 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.459496 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.459508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.459523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.459554 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.462243 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.462984 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.462992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.462994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.463130 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.463527 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.463531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.463533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.463880 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.464207 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.464527 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.464811 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 11:17:03.471099 9400 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:03.469251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.2.223.129:34847)
0807 11:17:03.469402 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.469407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.469769 (+ 362us) server_negotiation.cc:407] Connection header received
0807 11:17:03.469898 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.469901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.469952 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.470029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.470858 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60346: BlockingRecv error: recv got EOF from 127.0.0.1:60346 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 11:17:03.471457 2942 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.473029 2942 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.474400 2942 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:17:03.488261 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.475818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.2.223.129:34847)
0807 11:17:03.476016 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.476022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.476166 (+ 144us) server_negotiation.cc:407] Connection header received
0807 11:17:03.476288 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.476291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.476346 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.476426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.477592 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.478277 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.479505 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.479915 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.480005 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.483353 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.483359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.483362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.483394 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.485890 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.486546 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.486550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.486552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.486667 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.487003 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.487006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.487008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.487260 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.487471 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.487822 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.488000 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:17:03.493100 9400 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:03.491144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.2.223.129:34847)
0807 11:17:03.491353 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.491361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.491645 (+ 284us) server_negotiation.cc:407] Connection header received
0807 11:17:03.491848 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.491853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.491928 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.492073 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.492843 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60360: BlockingRecv error: recv got EOF from 127.0.0.1:60360 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 11:17:03.493489 2942 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786101369622807-2942-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.494987 2942 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:03.496227 2942 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:17:03.510028 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.497405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.2.223.129:34847)
0807 11:17:03.497640 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.497647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.497858 (+ 211us) server_negotiation.cc:407] Connection header received
0807 11:17:03.497975 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.497978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.498029 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.498108 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.499083 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.499919 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.501071 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.501475 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.501570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.505136 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.505141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.505144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.505175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.507699 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.508300 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.508304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.508307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.508437 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.508765 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.508768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.508770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.509020 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.509261 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.509496 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.509729 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:17:03.511713 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8970
I20260807 11:17:03.523727 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.512380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.2.223.129:34847)
0807 11:17:03.512596 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.512603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.512975 (+ 372us) server_negotiation.cc:407] Connection header received
0807 11:17:03.513194 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.513199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.513269 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.513370 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.514437 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.515131 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.516312 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.516749 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.516895 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.518907 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.518914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.518918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.518954 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.521449 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.522035 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.522038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.522040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.522172 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.522469 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.522472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.522474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.522768 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.522980 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.523234 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.523426 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:17:03.524289 9098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:03.608784 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8970
I20260807 11:17:03.629984 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9109
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:17:03.645282 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.630883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.2.223.130:34929)
0807 11:17:03.631457 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.631464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.631491 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:03.631570 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.631577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.631810 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.631948 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.632894 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.633896 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.635406 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.635955 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.636054 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.639893 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.639904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.639917 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.639949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.642784 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.643441 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.643448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.643451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.643570 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.643911 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.643914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.643916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.644234 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.644458 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.644805 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.644994 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260807 11:17:03.646696 9241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:03.737529 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9109
I20260807 11:17:03.758342 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9252
Aug 07 11:17:03 dist-test-slave-5l3k krb5kdc[8856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:17:03.776038 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:03.759395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.2.223.131:44585)
0807 11:17:03.760030 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:03.760037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:03.760069 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:17:03.760133 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:03.760140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:03.760368 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:03.760557 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:03.761742 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.762988 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.764344 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:03.764953 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:03.765111 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:03.769656 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:03.769668 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:03.769684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:03.769719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:03.772843 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.773633 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.773648 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.773651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.773787 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:03.774251 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:03.774257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:03.774261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:03.774715 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:03.775078 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:03.775407 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 11:17:03.775713 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":316,"threads_started":1}
I20260807 11:17:03.777432 9384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:03.863507 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9252
I20260807 11:17:03.884222 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 8869
I20260807 11:17:03.885519 8939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:03.993700 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 8869
2026-08-07T11:17:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4185 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:04 dist-test-slave-5l3k krb5kdc[9422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:04 dist-test-slave-5l3k krb5kdc[9422](info): set up 2 sockets
Aug 07 11:17:04 dist-test-slave-5l3k krb5kdc[9422](info): commencing operation
krb5kdc: starting...
W20260807 11:17:05.140170 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:05 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab.
Entry for principal oryx/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab.
I20260807 11:17:05.388062 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40979
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab
--principal=oryx/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:05.720357 9436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:05.720723 9436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:05.720890 9436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:05.728154 9436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:05.728295 9436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:05.728410 9436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:05.728529 9436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:05.728615 9436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:05.738422 9436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab
--principal=oryx/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:05.739934 9436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:05.741387 9436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:05.754052 9441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:05.754096 9442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:05.754155 9444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:05.755134 9436 server_base.cc:1034] running on GCE node
I20260807 11:17:05.756022 9436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:05.757488 9436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:05.758742 9436 hybrid_clock.cc:648] HybridClock initialized: now 1786101425758689 us; error 60 us; skew 500 ppm
I20260807 11:17:05.759191 9436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:05 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:05.762594 9436 init.cc:377] Logged in from keytab as oryx/127.2.223.190@KRBTEST.COM (short username oryx)
I20260807 11:17:05.764159 9436 webserver.cc:460] Webserver started at http://127.2.223.190:39301/ using document root <none> and password file <none>
I20260807 11:17:05.764884 9436 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:05.765039 9436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:05.765383 9436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:05.768535 9436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "7eb9c2844610439d943d7fd409bc01af"
format_stamp: "Formatted at 2026-08-07 11:17:05 on dist-test-slave-5l3k"
server_key: "0e5b724d28aaa669997e3c347ba12276"
server_key_iv: "d2d39e41c4f9549bb817908c36e3f4b4"
server_key_version: "encryptionkey@0"
I20260807 11:17:05.769310 9436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "7eb9c2844610439d943d7fd409bc01af"
format_stamp: "Formatted at 2026-08-07 11:17:05 on dist-test-slave-5l3k"
server_key: "0e5b724d28aaa669997e3c347ba12276"
server_key_iv: "d2d39e41c4f9549bb817908c36e3f4b4"
server_key_version: "encryptionkey@0"
I20260807 11:17:05.774691 9436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:05.778254 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:05.779649 9436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:05.779834 9436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "7eb9c2844610439d943d7fd409bc01af"
format_stamp: "Formatted at 2026-08-07 11:17:05 on dist-test-slave-5l3k"
server_key: "0e5b724d28aaa669997e3c347ba12276"
server_key_iv: "d2d39e41c4f9549bb817908c36e3f4b4"
server_key_version: "encryptionkey@0"
I20260807 11:17:05.780072 9436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:05.793128 9436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:05.797014 9436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:05.797308 9436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:05.816642 9436 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:40979
I20260807 11:17:05.816687 9506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:40979 every 8 connection(s)
I20260807 11:17:05.818120 9436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:05.820454 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9436
I20260807 11:17:05.820696 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:05.821601 2942 external_mini_cluster.cc:1428] Setting key 2471586702808c43b354161e518b085c
I20260807 11:17:05.822863 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:05.834872 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Bootstrap starting.
Aug 07 11:17:05 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:05.838304 9507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:05.839401 9507 log.cc:824] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:05.842213 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: No bootstrap required, opened a new log
I20260807 11:17:05.843598 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:05.823753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.2.223.190:40979)
0807 11:17:05.824500 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:05.824511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:05.824552 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:17:05.826119 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:05.826140 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:05.826553 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:05.826985 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:05.828452 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:05.829718 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:05.831014 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:05.831722 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:05.831951 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:05.836118 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:05.836126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:05.836142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:05.836186 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:05.839441 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:05.840200 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:05.840211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:05.840227 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:05.840407 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:05.840868 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:05.840873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:05.840878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:05.841390 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:05.841735 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:05.842145 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 11:17:05.842609 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":322,"threads_started":1}
I20260807 11:17:05.850765 9507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } }
I20260807 11:17:05.851128 9507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:05.851233 9507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb9c2844610439d943d7fd409bc01af, State: Initialized, Role: FOLLOWER
I20260807 11:17:05.851805 9507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } }
I20260807 11:17:05.852046 9507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:05.852180 9507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:05.852380 9507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:05.854692 9507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } }
I20260807 11:17:05.855106 9507 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb9c2844610439d943d7fd409bc01af; no voters:
I20260807 11:17:05.855634 9507 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:05.855795 9512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:05.856554 9512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 LEADER]: Becoming Leader. State: Replica: 7eb9c2844610439d943d7fd409bc01af, State: Running, Role: LEADER
I20260807 11:17:05.857133 9512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } }
I20260807 11:17:05.857647 9507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:05.859884 9513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } }
I20260807 11:17:05.860280 9513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:05.860853 9514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb9c2844610439d943d7fd409bc01af. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } }
I20260807 11:17:05.861172 9514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:05.861152 9521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:05.867357 9521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:05.876732 9521 catalog_manager.cc:1347] Generated new cluster ID: 1c6f8d012925476e84ca5d854f65e06d
I20260807 11:17:05.876883 9521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:05.890584 9521 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:05.891827 9521 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:05.909700 9521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Generated new TSK 0
I20260807 11:17:05.910897 9521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
I20260807 11:17:05.978448 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:06.329471 9533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:06.329764 9533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:06.329864 9533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:06.337082 9533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:06.337201 9533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:06.337319 9533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:06.347517 9533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:06.348951 9533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:06.350306 9533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:06.364742 9539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:06.364728 9538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:06.364773 9541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:06.365681 9533 server_base.cc:1034] running on GCE node
I20260807 11:17:06.366374 9533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:06.367408 9533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:06.368664 9533 hybrid_clock.cc:648] HybridClock initialized: now 1786101426368603 us; error 55 us; skew 500 ppm
I20260807 11:17:06.369005 9533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:06 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101426, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:06.372360 9533 init.cc:377] Logged in from keytab as oryx/127.2.223.129@KRBTEST.COM (short username oryx)
I20260807 11:17:06.374125 9533 webserver.cc:460] Webserver started at http://127.2.223.129:35955/ using document root <none> and password file <none>
I20260807 11:17:06.374928 9533 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:06.375078 9533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:06.375402 9533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:06.378412 9533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "2c2dae56828b4222b2f13b759e771dfd"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "4bd814cdb35fb46e7de7777fdd02fd6a"
server_key_iv: "1e931a1900d705b9bd7c08acdddaaffd"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.379281 9533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "2c2dae56828b4222b2f13b759e771dfd"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "4bd814cdb35fb46e7de7777fdd02fd6a"
server_key_iv: "1e931a1900d705b9bd7c08acdddaaffd"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.384862 9533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:06.388496 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:06.389873 9533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:06.390053 9533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "2c2dae56828b4222b2f13b759e771dfd"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "4bd814cdb35fb46e7de7777fdd02fd6a"
server_key_iv: "1e931a1900d705b9bd7c08acdddaaffd"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.390281 9533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:06.432991 9533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:06.437273 9533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:06.437528 9533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:06.438544 9533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:06.439963 9533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:06.440038 9533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:06.440132 9533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:06.440178 9533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:06.475793 9533 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:37595
I20260807 11:17:06.475896 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:37595 every 8 connection(s)
I20260807 11:17:06.477151 9533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:06.485656 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9533
I20260807 11:17:06.485904 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:06.486630 2942 external_mini_cluster.cc:1428] Setting key 61f23ee799759e4457cd5d55f728d740
Aug 07 11:17:06 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101426, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:06.499071 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:06.480088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:51249 (local address 127.2.223.190:40979)
0807 11:17:06.480581 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:06.480586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:06.481643 (+ 1057us) server_negotiation.cc:407] Connection header received
0807 11:17:06.483678 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:06.483682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:06.483755 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:06.483874 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:06.485595 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:06.486457 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:06.488349 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:06.488759 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:06.488902 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:06.492829 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:06.492834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:06.492837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:06.492871 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:06.495782 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:06.496568 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:06.496574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:06.496578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:06.496716 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:06.497191 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:06.497197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:06.497200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:06.497484 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:06.497667 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:06.498417 (+ 750us) server_negotiation.cc:299] Negotiation successful
0807 11:17:06.498695 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260807 11:17:06.500121 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:06.480706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:51249)
0807 11:17:06.481458 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:06.481506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:06.483316 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:06.484124 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:06.484132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:06.484803 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:06.485401 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:06.485419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:06.486679 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:06.486684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:06.488180 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:06.488194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:06.489019 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:06.489026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:06.489299 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:06.490029 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:06.490053 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:06.492585 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:06.495949 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:06.495958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:06.495984 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:06.496431 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:06.496845 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:06.496850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:06.496854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:06.497064 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:06.497621 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:06.497630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:06.498025 (+ 395us) client_negotiation.cc:769] Sending connection context
0807 11:17:06.498317 (+ 292us) client_negotiation.cc:240] Negotiation successful
0807 11:17:06.498922 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260807 11:17:06.501801 9667 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:06.502127 9667 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:06.502978 9667 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:06.505439 9470 ts_manager.cc:194] Registered new tserver with Master: 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:06.507732 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:51249
WARNING: no policy specified for oryx/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
I20260807 11:17:06.548118 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:06.890271 9673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:06.890697 9673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:06.890929 9673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:06.898667 9673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:06.898784 9673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:06.898913 9673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:06.909039 9673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:06.910419 9673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:06.912020 9673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:06.928251 9678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:06.928300 9679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:06.928345 9681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:06.932018 9673 server_base.cc:1034] running on GCE node
I20260807 11:17:06.932723 9673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:06.933689 9673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:06.934928 9673 hybrid_clock.cc:648] HybridClock initialized: now 1786101426934874 us; error 45 us; skew 500 ppm
I20260807 11:17:06.935262 9673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:06 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101426, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:06.938407 9673 init.cc:377] Logged in from keytab as oryx/127.2.223.130@KRBTEST.COM (short username oryx)
I20260807 11:17:06.939986 9673 webserver.cc:460] Webserver started at http://127.2.223.130:39377/ using document root <none> and password file <none>
I20260807 11:17:06.940739 9673 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:06.940891 9673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:06.941232 9673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:06.944380 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "defc27725dbf4ab08e4f5646b5bc1de6"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "d6f9184d0453fcf4a18fbb27adc39a7b"
server_key_iv: "71f3909f3d85ae200697042f7b43a80c"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.945266 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "defc27725dbf4ab08e4f5646b5bc1de6"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "d6f9184d0453fcf4a18fbb27adc39a7b"
server_key_iv: "71f3909f3d85ae200697042f7b43a80c"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.950744 9673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:17:06.954180 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:06.955588 9673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:06.955766 9673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "defc27725dbf4ab08e4f5646b5bc1de6"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "d6f9184d0453fcf4a18fbb27adc39a7b"
server_key_iv: "71f3909f3d85ae200697042f7b43a80c"
server_key_version: "encryptionkey@0"
I20260807 11:17:06.955981 9673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:06.989480 9673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:06.994020 9673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:06.994400 9673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:06.995559 9673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:06.996956 9673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:06.997064 9673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:06.997229 9673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:06.997304 9673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:07.033879 9673 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:33053
I20260807 11:17:07.034039 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:33053 every 8 connection(s)
I20260807 11:17:07.035274 9673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:07.044708 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9673
I20260807 11:17:07.044919 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:07.045413 2942 external_mini_cluster.cc:1428] Setting key fcd332672e79d6de8ba5910d87e9b051
Aug 07 11:17:07 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101426, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:07.056818 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:07.037970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:55797 (local address 127.2.223.190:40979)
0807 11:17:07.038528 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:07.038535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:07.039502 (+ 967us) server_negotiation.cc:407] Connection header received
0807 11:17:07.041344 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:07.041348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:07.041418 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:07.041518 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:07.043049 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.043770 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.045406 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.046024 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.046146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:07.049745 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:07.049754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:07.049763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:07.049806 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:07.053571 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:07.054277 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:07.054285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:07.054291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:07.054453 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:07.054917 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:07.054924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:07.054929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:07.055314 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:07.055472 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:07.056093 (+ 621us) server_negotiation.cc:299] Negotiation successful
0807 11:17:07.056345 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":238,"threads_started":1}
I20260807 11:17:07.057523 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:07.038593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:55797)
0807 11:17:07.039338 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:07.039377 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:07.041075 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:07.041724 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:07.041732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:07.042269 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:07.042902 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:07.042920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.043893 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.043897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:07.045260 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:07.045272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.046270 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.046286 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:07.046517 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:07.047250 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:07.047289 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:07.049511 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:07.053713 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:07.053721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:07.053733 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:07.054144 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:07.054601 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:07.054604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:07.054606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:07.054785 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:07.055432 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:07.055437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:07.055738 (+ 301us) client_negotiation.cc:769] Sending connection context
0807 11:17:07.055957 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 11:17:07.056329 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":309,"threads_started":1}
I20260807 11:17:07.059221 9810 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:07.059613 9810 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:07.060420 9810 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:07.062047 9470 ts_manager.cc:194] Registered new tserver with Master: defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053)
I20260807 11:17:07.063429 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:55797
WARNING: no policy specified for oryx/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
I20260807 11:17:07.106230 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:07.449151 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:07.449451 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:07.449561 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:07.456872 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:07.456984 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:07.457099 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:07.467279 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:07.468576 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:07.469903 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:07.485401 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:07.485484 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:07.485538 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:07.486770 9816 server_base.cc:1034] running on GCE node
I20260807 11:17:07.487370 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:07.488363 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:07.489595 9816 hybrid_clock.cc:648] HybridClock initialized: now 1786101427489533 us; error 57 us; skew 500 ppm
I20260807 11:17:07.489905 9816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:07 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101427, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:07.499480 9816 init.cc:377] Logged in from keytab as oryx/127.2.223.131@KRBTEST.COM (short username oryx)
I20260807 11:17:07.500921 9816 webserver.cc:460] Webserver started at http://127.2.223.131:42449/ using document root <none> and password file <none>
I20260807 11:17:07.501590 9816 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:07.501693 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:07.501940 9816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:07.504954 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "495ff7334ca549d9a4a788d2071cb08f"
format_stamp: "Formatted at 2026-08-07 11:17:07 on dist-test-slave-5l3k"
server_key: "15b07f5cfec2b99e0a57b9448cc2265f"
server_key_iv: "350e935c56906c2c4285f84aabb56ac7"
server_key_version: "encryptionkey@0"
I20260807 11:17:07.505677 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "495ff7334ca549d9a4a788d2071cb08f"
format_stamp: "Formatted at 2026-08-07 11:17:07 on dist-test-slave-5l3k"
server_key: "15b07f5cfec2b99e0a57b9448cc2265f"
server_key_iv: "350e935c56906c2c4285f84aabb56ac7"
server_key_version: "encryptionkey@0"
I20260807 11:17:07.510941 9667 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
I20260807 11:17:07.511008 9816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:17:07.514635 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:07.516012 9816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:07.516186 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "495ff7334ca549d9a4a788d2071cb08f"
format_stamp: "Formatted at 2026-08-07 11:17:07 on dist-test-slave-5l3k"
server_key: "15b07f5cfec2b99e0a57b9448cc2265f"
server_key_iv: "350e935c56906c2c4285f84aabb56ac7"
server_key_version: "encryptionkey@0"
I20260807 11:17:07.516399 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:07.532270 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:07.536085 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:07.536376 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:07.537317 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:07.538733 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:07.538807 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:07.538981 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:07.539036 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:07.574209 9816 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37973
I20260807 11:17:07.574257 9951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37973 every 8 connection(s)
I20260807 11:17:07.575582 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:07.583546 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9816
I20260807 11:17:07.583738 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:07.584367 2942 external_mini_cluster.cc:1428] Setting key 3f9a5576d4e893b4207d936ea6e80c75
Aug 07 11:17:07 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101427, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:07.597131 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:07.578301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:55201 (local address 127.2.223.190:40979)
0807 11:17:07.578818 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:07.578823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:07.579865 (+ 1042us) server_negotiation.cc:407] Connection header received
0807 11:17:07.581814 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:07.581820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:07.581932 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:07.582073 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:07.583963 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.585145 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.587033 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.587618 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.587765 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:07.591328 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:07.591333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:07.591336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:07.591375 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:07.593887 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:07.594693 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:07.594698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:07.594702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:07.594840 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:07.595332 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:07.595337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:07.595342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:07.595627 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:07.595789 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:07.596495 (+ 706us) server_negotiation.cc:299] Negotiation successful
0807 11:17:07.596735 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
I20260807 11:17:07.598111 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:07.578847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:55201)
0807 11:17:07.579671 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:07.579713 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:07.581505 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:07.582336 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:07.582345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:07.582964 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:07.583687 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:07.583718 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.585322 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.585329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:07.586881 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:07.586895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:07.587867 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:07.587877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:07.588134 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:07.588840 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:07.588863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:07.591086 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:07.594037 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:07.594046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:07.594063 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:07.594563 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:07.594959 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:07.594982 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:07.594986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:07.595204 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:07.595764 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:07.595772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:07.596130 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 11:17:07.596405 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 11:17:07.596950 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260807 11:17:07.599634 9952 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:07.599946 9952 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:07.600693 9952 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:07.602147 9470 ts_manager.cc:194] Registered new tserver with Master: 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:07.603025 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:55201
I20260807 11:17:07.612388 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.223.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.189.keytab.
Entry for principal oryx/127.2.223.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.189.keytab.
I20260807 11:17:07.674052 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:37579
--webserver_interface=127.2.223.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.189.keytab
--principal=oryx/127.2.223.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40979,127.2.223.189:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:08.020228 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:08.020570 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:08.020747 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:08.028074 9958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:08.028173 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:08.028228 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:08.028288 9958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:08.028332 9958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:08.038154 9958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40979,127.2.223.189:37579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.189.keytab
--principal=oryx/127.2.223.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.223.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
W20260807 11:17:08.038542 9958 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.223.190:40979,127.2.223.189:37579'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 11:17:08.039479 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:08.040766 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:08.052762 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.052940 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.052997 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:08.053970 9958 server_base.cc:1034] running on GCE node
I20260807 11:17:08.054805 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:08.056067 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:08.057304 9958 hybrid_clock.cc:648] HybridClock initialized: now 1786101428057234 us; error 49 us; skew 500 ppm
I20260807 11:17:08.057713 9958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:08.061321 9958 init.cc:377] Logged in from keytab as oryx/127.2.223.189@KRBTEST.COM (short username oryx)
I20260807 11:17:08.062736 9958 webserver.cc:460] Webserver started at http://127.2.223.189:37893/ using document root <none> and password file <none>
I20260807 11:17:08.063373 9958 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:08.063480 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:08.063910 9958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:08.065991 9810 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
I20260807 11:17:08.068228 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/instance:
uuid: "9a820e97b0704c258ffad09d42418ca2"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "5c3c0156e7d0a094c99b5ad421bb2d77"
server_key_iv: "99ea86591085a568336a5a109e44eefe"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.069166 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal/instance:
uuid: "9a820e97b0704c258ffad09d42418ca2"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "5c3c0156e7d0a094c99b5ad421bb2d77"
server_key_iv: "99ea86591085a568336a5a109e44eefe"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.075627 9958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 11:17:08.080363 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:08.082011 9958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:17:08.082252 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "9a820e97b0704c258ffad09d42418ca2"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "5c3c0156e7d0a094c99b5ad421bb2d77"
server_key_iv: "99ea86591085a568336a5a109e44eefe"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.082584 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:08.103392 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:08.109323 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:08.109649 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:08.159950 9958 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.189:37579
I20260807 11:17:08.160054 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.189:37579 every 8 connection(s)
I20260807 11:17:08.161998 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
I20260807 11:17:08.167382 10030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:08.170007 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 9958
I20260807 11:17:08.179895 10030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.189@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:08.205325 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.172730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.2.223.189:37579)
0807 11:17:08.174100 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.174112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.174190 (+ 78us) server_negotiation.cc:407] Connection header received
0807 11:17:08.176924 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.176946 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.177421 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.177919 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.180570 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.182376 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.185417 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.186663 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.186944 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.193814 (+ 6870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.193829 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.193846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.193917 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.199370 (+ 5453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.200411 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.200424 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.200434 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.200793 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.201417 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.201425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.201430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.202422 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.202952 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.203284 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.203788 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":729,"threads_started":1}
I20260807 11:17:08.208232 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.183102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44144)
0807 11:17:08.185223 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.185283 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.185506 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.186057 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.186067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.187529 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.188313 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.188325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.189789 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.189793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.191419 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.191431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.192746 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.192757 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.193076 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.194139 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.194220 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.198875 (+ 4655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.204376 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.204385 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.204391 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.205058 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.205673 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.205680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.205684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.205914 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.206863 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.206869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.207226 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.207421 (+ 195us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.207723 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":1247,"threads_started":1}
I20260807 11:17:08.208329 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.182955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.2.223.190:40979)
0807 11:17:08.183866 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.183875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.185624 (+ 1749us) server_negotiation.cc:407] Connection header received
0807 11:17:08.185690 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.185693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.185806 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.185899 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.188476 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.189603 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.191612 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.192352 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.192468 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.199138 (+ 6670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.199147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.199154 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.199201 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.204229 (+ 5028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.205245 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.205251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.205254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.205499 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.206079 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.206084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.206087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.206712 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.206868 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.207588 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.207823 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":572,"threads_started":1}
I20260807 11:17:08.210273 10030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.189@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
I20260807 11:17:08.227960 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.210946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43888)
0807 11:17:08.211111 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.211128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.211208 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.211495 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.211498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.212072 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.212325 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.212334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.213753 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.213756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.215075 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.215087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.216032 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.216036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.216310 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.217001 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.217025 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.220255 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.225090 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.225094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.225097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.225633 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.226055 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.226058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.226061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.226251 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.227007 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.227012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.227185 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.227303 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.227596 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 11:17:08.228260 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.211126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.2.223.189:37579)
0807 11:17:08.211274 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.211278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.211293 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:17:08.211331 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.211334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.211385 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.211463 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.212475 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.213593 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.215223 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.215892 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.216006 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.220500 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.220506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.220510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.220542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.224927 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.225749 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.225753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.225756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.225951 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.226352 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.226355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.226358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.226887 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.227012 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.227442 (+ 430us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.227748 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":53}
I20260807 11:17:08.234957 10030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Bootstrap starting.
I20260807 11:17:08.239603 10030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:08.241523 10030 log.cc:824] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:08.245317 10030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: No bootstrap required, opened a new log
I20260807 11:17:08.257944 10030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } }
I20260807 11:17:08.258577 10030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:08.258792 10030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a820e97b0704c258ffad09d42418ca2, State: Initialized, Role: FOLLOWER
I20260807 11:17:08.259747 10030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } }
I20260807 11:17:08.261492 10037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } } }
I20260807 11:17:08.262183 10037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:08.262732 10030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:17:08.280349 10048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:17:08.280647 10048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:17:08.284195 9958 master_runner.cc:428] Detected that this master 9a820e97b0704c258ffad09d42418ca2 is joining an existing cluster
I20260807 11:17:08.284369 9958 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.223.189:37579
I20260807 11:17:08.310576 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.294522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43892)
0807 11:17:08.295931 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.295946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.296101 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.296431 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.296435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.297089 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.297408 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.297418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.298883 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.298888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.300104 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.300116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.301183 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.301191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.301379 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.302024 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.302046 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.302917 (+ 871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.307358 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.307364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.307370 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.307972 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.308423 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.308429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.308433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.308616 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.309287 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.309293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.309455 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.309626 (+ 171us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.309954 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"thread_start_us":1016,"threads_started":1}
I20260807 11:17:08.310602 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.294864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.2.223.189:37579)
0807 11:17:08.295115 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.295123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.296035 (+ 912us) server_negotiation.cc:407] Connection header received
0807 11:17:08.296230 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.296234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.296304 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.296402 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.297553 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.298673 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.300247 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.300988 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.301140 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.303306 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.303310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.303313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.303401 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.307213 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.308113 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.308119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.308123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.308289 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.308739 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.308745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.308750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.309135 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.309319 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.309742 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.310102 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":77,"spinlock_wait_cycles":17280}
I20260807 11:17:08.312510 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.292439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.2.223.190:40979)
0807 11:17:08.292721 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.292729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.294385 (+ 1656us) server_negotiation.cc:407] Connection header received
0807 11:17:08.294684 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.294692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.294819 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.294900 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.297461 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.298733 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.301252 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.301986 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.302115 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.305084 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.305091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.305099 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.305147 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.308532 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.309324 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.309330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.309336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.309474 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.309903 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.309910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.309913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.310308 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.310469 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.311906 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.312152 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:17:08.313500 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.293003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44148)
0807 11:17:08.294253 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.294278 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.294452 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.295223 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.295228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.296730 (+ 1502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.297238 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.297253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.299000 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.299005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.301038 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.301059 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.302463 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.302472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.302704 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.303346 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.303370 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.304845 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.308658 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.308663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.308667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.309169 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.309633 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.309637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.309640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.309804 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.310561 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.310568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.311555 (+ 987us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.312946 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.313200 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":3,"thread_start_us":786,"threads_started":1}
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.190@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.223.188@KRBTEST.COM; defaulting to no policy
I20260807 11:17:08.341907 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.319284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43898)
0807 11:17:08.323051 (+ 3767us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.323092 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.323310 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.323817 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.323825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.324657 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.324912 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.324922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.326172 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.326178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.328228 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.328246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.329566 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.329576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.329802 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.330637 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.330666 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.333952 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.338584 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.338600 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.338610 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.339253 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.339765 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.339772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.339778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.339959 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.340671 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.340683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.340921 (+ 238us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.341144 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.341544 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3535,"thread_start_us":3379,"threads_started":1}
I20260807 11:17:08.342105 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.319304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.2.223.189:37579)
0807 11:17:08.319492 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.319501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.323180 (+ 3679us) server_negotiation.cc:407] Connection header received
0807 11:17:08.323537 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.323541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.323646 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.323762 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.325084 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.325997 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.328404 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.329365 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.329538 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.334191 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.334199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.334205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.334247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.338400 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.339422 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.339429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.339433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.339621 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.340113 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.340121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.340125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.340504 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.340671 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.341283 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.341701 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:17:08.344066 9470 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.223.189:37579: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9a820e97b0704c258ffad09d42418ca2"
member_type: NON_VOTER
last_known_addr {
host: "127.2.223.189"
port: 37579
}
attrs {
promote: true
}
}
dest_uuid: "7eb9c2844610439d943d7fd409bc01af"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.188@KRBTEST.COM" created.
I20260807 11:17:08.345518 9470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } }
W20260807 11:17:08.362753 9980 tablet.cc:2369] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:17:08.367453 10061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } }
I20260807 11:17:08.368640 10004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:08.374171 10061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.223.189:37579
I20260807 11:17:08.374506 10004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 11:17:08.375218 9958 master.cc:537] Master@127.2.223.189:37579 shutting down...
I20260807 11:17:08.375222 10062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } }
I20260807 11:17:08.376273 10062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:08.376710 10061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:17:08.390583 9455 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260807 11:17:08.394621 9958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:08.395116 9958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:08.395378 9958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: stopping tablet replica
W20260807 11:17:08.396186 9455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af -> Peer 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189:37579): Couldn't send request to peer 9a820e97b0704c258ffad09d42418ca2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.188.keytab.
Entry for principal oryx/127.2.223.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.188.keytab.
I20260807 11:17:08.406857 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:33277
--webserver_interface=127.2.223.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.188.keytab
--principal=oryx/127.2.223.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:17:08.413322 9958 master.cc:552] Master@127.2.223.189:37579 shutdown complete.
I20260807 11:17:08.413483 9958 master_runner.cc:305] Clearing existing system tablet
I20260807 11:17:08.413947 9958 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:08.414074 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:08.416715 9958 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260807 11:17:08.418326 10069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:08.418716 9958 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:08.419178 9958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 11:17:08.419322 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "9a820e97b0704c258ffad09d42418ca2"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "5c3c0156e7d0a094c99b5ad421bb2d77"
server_key_iv: "99ea86591085a568336a5a109e44eefe"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.424813 9958 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:08.431860 9958 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 11:17:08.432109 9958 log.cc:1196] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 11:17:08.432626 9958 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Deleting consensus metadata
I20260807 11:17:08.432955 9958 master_runner.cc:315] Copying system tablet from 127.2.223.190:40979
I20260807 11:17:08.435949 9958 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.223.190:40979
I20260807 11:17:08.456379 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.437119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.2.223.190:40979)
0807 11:17:08.437321 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.437328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.441596 (+ 4268us) server_negotiation.cc:407] Connection header received
0807 11:17:08.441817 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.441826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.441932 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.442059 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:08.443216 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.444400 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.446261 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.446981 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.447113 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.449734 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:08.449740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:08.449744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:08.449784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:08.453319 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.454161 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.454168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.454171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.454327 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:08.454800 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:08.454805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:08.454808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:08.455174 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:08.455336 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.455797 (+ 461us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.456019 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:17:08.456739 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.437047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44150)
0807 11:17:08.441451 (+ 4404us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.441473 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.441645 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.442112 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.442117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.442694 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:08.443032 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.443045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.444559 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.444565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.446071 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.446094 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.447153 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.447159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.447333 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.448386 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:08.448410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:08.449500 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:08.453467 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.453473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.453477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.454009 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.454450 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:08.454455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:08.454460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:08.454666 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:08.455316 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:08.455324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:08.455499 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.455651 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.455977 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4259,"thread_start_us":4145,"threads_started":1}
I20260807 11:17:08.457994 9491 tablet_copy_service.cc:140] P 7eb9c2844610439d943d7fd409bc01af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9a820e97b0704c258ffad09d42418ca2 ({username='oryx', principal='oryx/127.2.223.189@KRBTEST.COM'} at 127.0.0.1:44150)
I20260807 11:17:08.458417 9491 tablet_copy_service.cc:161] P 7eb9c2844610439d943d7fd409bc01af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9a820e97b0704c258ffad09d42418ca2 at {username='oryx', principal='oryx/127.2.223.189@KRBTEST.COM'} at 127.0.0.1:44150: session id = 9a820e97b0704c258ffad09d42418ca2-00000000000000000000000000000000
I20260807 11:17:08.463461 9491 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:17:08.466267 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 11:17:08.474737 9958 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: tablet copy: Starting download of 0 data blocks...
I20260807 11:17:08.475165 9958 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: tablet copy: Starting download of 1 WAL segments...
I20260807 11:17:08.482126 9958 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:17:08.487213 9491 tablet_copy_service.cc:342] P 7eb9c2844610439d943d7fd409bc01af: Request end of tablet copy session 9a820e97b0704c258ffad09d42418ca2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.223.189@KRBTEST.COM'} at 127.0.0.1:44150
I20260807 11:17:08.487517 9491 tablet_copy_service.cc:433] P 7eb9c2844610439d943d7fd409bc01af: ending tablet copy session 9a820e97b0704c258ffad09d42418ca2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9a820e97b0704c258ffad09d42418ca2
I20260807 11:17:08.500063 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:08.502858 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.504325 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.510912 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:08.511796 9958 server_base.cc:1034] running on GCE node
I20260807 11:17:08.512197 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:08.512867 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:08.514158 9958 hybrid_clock.cc:648] HybridClock initialized: now 1786101428514094 us; error 34 us; skew 500 ppm
I20260807 11:17:08.514467 9958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:08.516921 9958 init.cc:377] Logged in from keytab as oryx/127.2.223.189@KRBTEST.COM (short username oryx)
I20260807 11:17:08.519397 9958 webserver.cc:460] Webserver started at http://127.2.223.189:33541/ using document root <none> and password file <none>
I20260807 11:17:08.519765 9958 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:08.519851 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:08.523833 9958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:17:08.526777 10087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:08.527380 9958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 11:17:08.527566 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "9a820e97b0704c258ffad09d42418ca2"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "5c3c0156e7d0a094c99b5ad421bb2d77"
server_key_iv: "99ea86591085a568336a5a109e44eefe"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.527803 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:08.561152 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:08.565402 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:08.605603 9952 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
I20260807 11:17:08.611845 9958 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.189:37579
I20260807 11:17:08.612859 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
I20260807 11:17:08.611845 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.189:37579 every 8 connection(s)
I20260807 11:17:08.620066 10144 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:17:08.622951 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Bootstrap starting.
I20260807 11:17:08.636380 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:17:08.636888 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: Bootstrap complete.
I20260807 11:17:08.637787 10144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } }
I20260807 11:17:08.638037 10144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a820e97b0704c258ffad09d42418ca2, State: Initialized, Role: LEARNER
I20260807 11:17:08.638304 10144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } }
I20260807 11:17:08.639242 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } }
I20260807 11:17:08.639554 10144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:08.639626 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:17:08.649770 10159 catalog_manager.cc:1259] Loaded cluster ID: 1c6f8d012925476e84ca5d854f65e06d
I20260807 11:17:08.649964 10159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: loading cluster ID for follower catalog manager: success
I20260807 11:17:08.652706 10159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: acquiring CA information for follower catalog manager: success
I20260807 11:17:08.654618 10159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 11:17:08.900995 10068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:08.901398 10068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:08.901624 10068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:08.911595 10068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:08.911836 10068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:08.911994 10068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:08.912209 10068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:08.912369 10068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:08.926602 10068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.188.keytab
--principal=oryx/127.2.223.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.223.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:08.928325 10068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:08.929850 10068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:08.947217 10164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.948138 10167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:08.948685 10165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:08.950080 10068 server_base.cc:1034] running on GCE node
I20260807 11:17:08.950978 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:08.953111 10068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:08.954448 10068 hybrid_clock.cc:648] HybridClock initialized: now 1786101428954353 us; error 63 us; skew 500 ppm
I20260807 11:17:08.954949 10068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:08 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:08.961889 10068 init.cc:377] Logged in from keytab as oryx/127.2.223.188@KRBTEST.COM (short username oryx)
I20260807 11:17:08.964455 10068 webserver.cc:460] Webserver started at http://127.2.223.188:39881/ using document root <none> and password file <none>
I20260807 11:17:08.965374 10068 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:08.965617 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:08.966058 10068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:08.967478 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.951692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.2.223.189:37579)
0807 11:17:08.952573 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:08.952583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:08.959048 (+ 6465us) server_negotiation.cc:407] Connection header received
0807 11:17:08.959292 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:08.959297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:08.959430 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:08.959590 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:08.960382 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.962445 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.964806 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.966023 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.966221 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.966561 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:08.966834 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 11:17:08.966992 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":434,"threads_started":1}
I20260807 11:17:08.967520 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:08.950993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43900)
0807 11:17:08.958902 (+ 7909us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:08.958930 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:08.959147 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:08.959752 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:08.959760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:08.959797 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:08.960037 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.960052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.962744 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.962750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.964636 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:08.964650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:08.966266 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:08.966272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:08.966629 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:08.966686 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 11:17:08.966796 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 11:17:08.966954 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7748,"thread_start_us":7632,"threads_started":1}
I20260807 11:17:08.969293 10117 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 11:17:08.970081 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb9c2844610439d943d7fd409bc01af. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } }
I20260807 11:17:08.970449 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:17:08.971045 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/instance:
uuid: "8b65b866775e4cdd81946944a14db9e8"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "dd79ea6f07542106fb29ba90785fa0a0"
server_key_iv: "301fead450684377ec2c06feffdc8de3"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.971978 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal/instance:
uuid: "8b65b866775e4cdd81946944a14db9e8"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "dd79ea6f07542106fb29ba90785fa0a0"
server_key_iv: "301fead450684377ec2c06feffdc8de3"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.978796 10068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 11:17:08.984090 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:08.985703 10068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260807 11:17:08.985958 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "8b65b866775e4cdd81946944a14db9e8"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "dd79ea6f07542106fb29ba90785fa0a0"
server_key_iv: "301fead450684377ec2c06feffdc8de3"
server_key_version: "encryptionkey@0"
I20260807 11:17:08.986318 10068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:09.012198 10068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:09.018415 10068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:09.018941 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:09.065718 10068 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.188:33277
I20260807 11:17:09.065872 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.188:33277 every 8 connection(s)
I20260807 11:17:09.067414 10068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
I20260807 11:17:09.071728 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 10068
I20260807 11:17:09.073318 10233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:09.087625 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.188@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:09.110772 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.074059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.2.223.188:33277)
0807 11:17:09.075988 (+ 1929us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.076008 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.076442 (+ 434us) server_negotiation.cc:407] Connection header received
0807 11:17:09.080396 (+ 3954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.080419 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.080890 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.081423 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.084362 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.086232 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.088830 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.089754 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.089882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.098676 (+ 8794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.098687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.098699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.098763 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.104832 (+ 6069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.105688 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.105695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.105701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.105941 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.106579 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.106583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.106585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.107270 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.107634 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.108289 (+ 655us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.108843 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1027,"threads_started":1}
I20260807 11:17:09.116708 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.091833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44160)
0807 11:17:09.093863 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.093933 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.094127 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.094713 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.094718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.095987 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.096635 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.096651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.099245 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.099251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.102076 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.102090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.103301 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.103312 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.103626 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.104533 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.104552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.108265 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.113371 (+ 5106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.113387 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.113392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.114049 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.114515 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.114522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.114529 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.114695 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.115309 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.115317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.115521 (+ 204us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.115737 (+ 216us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.116056 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"thread_start_us":1218,"threads_started":1}
I20260807 11:17:09.116798 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.091663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.2.223.190:40979)
0807 11:17:09.092856 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.092862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.094052 (+ 1190us) server_negotiation.cc:407] Connection header received
0807 11:17:09.094287 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.094296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.094405 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.094635 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.096878 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.098957 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.102228 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.102963 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.103122 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.108590 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.108598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.108603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.108646 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.113226 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.114175 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.114181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.114186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.114365 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.114814 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.114819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.114824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.115163 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.115369 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.115819 (+ 450us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.116089 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":646,"threads_started":1}
I20260807 11:17:09.118424 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.188@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
I20260807 11:17:09.134317 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.119118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43902)
0807 11:17:09.119390 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.119409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.119538 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.119867 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.119870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.120347 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.120512 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.120520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.121788 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.121792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.123105 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.123118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.124022 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.124026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.124207 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.124817 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.124833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.127883 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.132128 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.132132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.132134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.132495 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.132904 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.132907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.132909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.133027 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.133618 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.133621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.133731 (+ 110us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.133837 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.134029 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:17:09.134626 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.119411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.2.223.189:37579)
0807 11:17:09.119622 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.119627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.119644 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:17:09.119693 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.119697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.119755 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.119835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.120655 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.121675 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.123235 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.123894 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.123994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.128099 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.128104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.128108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.128138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.131995 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.132593 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.132597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.132599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.132807 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.133152 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.133155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.133157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.133520 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.133632 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.133924 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.134142 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:17:09.135557 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101428, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.188@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
I20260807 11:17:09.152338 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.136564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.2.223.188:33277)
0807 11:17:09.136728 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.136732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.137377 (+ 645us) server_negotiation.cc:407] Connection header received
0807 11:17:09.137533 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.137538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.137604 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.137700 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.138563 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.139418 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.141045 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.141690 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.141879 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.145607 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.145611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.145615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.145641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.149153 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.149952 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.149960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.149963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.150107 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.150649 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.150656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.150659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.150959 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.151111 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.151525 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.151910 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":69}
I20260807 11:17:09.152338 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.136998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33616)
0807 11:17:09.137292 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.137308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.137430 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.137716 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.137720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.138159 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.138402 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.138411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.139547 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.139555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.140908 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.140920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.141949 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.141955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.142288 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.142942 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.142957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.145434 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.149279 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.149286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.149290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.149830 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.150218 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.150224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.150227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.150532 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.151076 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.151081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.151275 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.151410 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.151690 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 11:17:09.159251 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Bootstrap starting.
I20260807 11:17:09.164006 10233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:09.166277 10233 log.cc:824] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:09.169507 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: No bootstrap required, opened a new log
I20260807 11:17:09.180361 10233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } }
I20260807 11:17:09.180770 10233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:09.180946 10233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b65b866775e4cdd81946944a14db9e8, State: Initialized, Role: FOLLOWER
I20260807 11:17:09.181497 10233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } }
I20260807 11:17:09.183143 10240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } } }
I20260807 11:17:09.183844 10240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:09.184104 10233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:17:09.201669 10251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:17:09.201967 10251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:17:09.206202 10068 master_runner.cc:428] Detected that this master 8b65b866775e4cdd81946944a14db9e8 is joining an existing cluster
I20260807 11:17:09.206527 10068 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.223.188:33277
I20260807 11:17:09.217903 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9533
I20260807 11:17:09.250825 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.214050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.2.223.188:33277)
0807 11:17:09.214288 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.214293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.223945 (+ 9652us) server_negotiation.cc:407] Connection header received
0807 11:17:09.224263 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.224269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.224356 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.224436 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.227062 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.228852 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.232781 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.234385 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.234543 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.240441 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.240449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.240454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.240490 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.245567 (+ 5077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.246779 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.246784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.246787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.246966 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.247494 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.247513 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.247516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.248240 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.248474 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.249418 (+ 944us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.249678 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":25344}
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.129@KRBTEST.COM
I20260807 11:17:09.254211 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.214736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33624)
0807 11:17:09.223814 (+ 9078us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.223835 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.224115 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.224707 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.224712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.225954 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.226821 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.226846 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.229307 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.229313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.232603 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.232622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.235192 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.235201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.235521 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.237449 (+ 1928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.237472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.239893 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.245799 (+ 5906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.245804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.245808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.246659 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.247160 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.247163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.247167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.247379 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.248552 (+ 1173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.248556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.248867 (+ 311us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.253090 (+ 4223us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.253480 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8891,"mutex_wait_us":940,"thread_start_us":16207,"threads_started":1}
I20260807 11:17:09.262001 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.230269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.2.223.189:37579)
0807 11:17:09.230571 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.230578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.236978 (+ 6400us) server_negotiation.cc:407] Connection header received
0807 11:17:09.237350 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.237356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.237475 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.237571 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.242373 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.243980 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.246893 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.248116 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.248262 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.252047 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.252053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.252057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.252111 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.257240 (+ 5129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.258258 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.258265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.258269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.258584 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.259326 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.259331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.259335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.259886 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.260011 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.261151 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.261453 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:17:09.263559 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.230702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43916)
0807 11:17:09.236702 (+ 6000us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.236722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.237135 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.237864 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.237870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.241136 (+ 3266us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.242159 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.242173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.244195 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.244200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.246710 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.246725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.248848 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.248859 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.249162 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.250213 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.250240 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.251822 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.257403 (+ 5581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.257408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.257412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.258101 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.258761 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.258766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.258771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.259109 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.260109 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.260115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.260626 (+ 511us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.262299 (+ 1673us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.262719 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5608,"mutex_wait_us":3,"thread_start_us":5792,"threads_started":1}
W20260807 11:17:09.271991 10184 tablet.cc:2369] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:17:09.272589 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.214259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.2.223.190:40979)
0807 11:17:09.214449 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.214453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.231935 (+ 17482us) server_negotiation.cc:407] Connection header received
0807 11:17:09.232128 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.232136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.232256 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.232373 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.235897 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.237614 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.257357 (+ 19743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.258433 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.258678 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.262588 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.262614 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.262620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.262685 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.268527 (+ 5842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.269442 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.269448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.269453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.269754 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.270383 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.270388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.270392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.270826 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.270998 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.271828 (+ 830us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.272143 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:17:09.273906 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.221030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.2.223.129:37595)
0807 11:17:09.236333 (+ 15303us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.236343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.236375 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:17:09.236494 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.236517 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.236863 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.240106 (+ 3243us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.240570 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.242669 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.245093 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.246183 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.256281 (+ 10098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.256809 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.256824 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.256844 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.256889 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.263178 (+ 6289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.269548 (+ 6370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.269557 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.269571 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.269846 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.270583 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.270590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.270593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.271219 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.273060 (+ 1841us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.273195 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.273411 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15084,"thread_start_us":15950,"threads_started":1}
I20260807 11:17:09.274058 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.214882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44166)
0807 11:17:09.231573 (+ 16691us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.231593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.231965 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.232668 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.232676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.234360 (+ 1684us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.235602 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.235627 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.237891 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.237899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.257157 (+ 19258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.257172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.259152 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.259160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.259499 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.260337 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.260358 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.262285 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.268690 (+ 6405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.268702 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.268707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.269285 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.269950 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.269971 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.269974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.270223 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.271113 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.271118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.271408 (+ 290us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.272870 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.273270 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16446,"mutex_wait_us":1927,"spinlock_wait_cycles":25472,"thread_start_us":14465,"threads_started":1}
I20260807 11:17:09.275880 9661 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.190@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
I20260807 11:17:09.305114 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.281418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.2.223.188:33277)
0807 11:17:09.281627 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.281634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.281923 (+ 289us) server_negotiation.cc:407] Connection header received
0807 11:17:09.282016 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.282021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.282166 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.282266 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.283763 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.285372 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.288094 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.289305 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.289453 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.295679 (+ 6226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.295686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.295691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.295729 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.300990 (+ 5261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.302153 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.302160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.302164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.302395 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.303114 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.303119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.303123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.303530 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.303697 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.304304 (+ 607us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.304576 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 11:17:09.305181 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.281261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33626)
0807 11:17:09.281629 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.281649 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.281807 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.282355 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.282360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.283159 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.283556 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.283589 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.285593 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.285602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.287838 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.287856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.289576 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.289634 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.290127 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.291044 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.291071 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.295382 (+ 4311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.301146 (+ 5764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.301152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.301157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.301996 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.302560 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.302567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.302571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.302940 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.303674 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.303680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.303936 (+ 256us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.304172 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.304540 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260807 11:17:09.307453 9470 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.223.188:33277: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8b65b866775e4cdd81946944a14db9e8"
member_type: NON_VOTER
last_known_addr {
host: "127.2.223.188"
port: 33277
}
attrs {
promote: true
}
}
dest_uuid: "7eb9c2844610439d943d7fd409bc01af"
cas_config_opid_index: 5
I20260807 11:17:09.308698 9470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } }
I20260807 11:17:09.311440 10169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.315460 10117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 11:17:09.317432 10206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:09.330049 10169 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.223.188:33277
I20260807 11:17:09.331257 10206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 11:17:09.332252 10068 master.cc:537] Master@127.2.223.188:33277 shutting down...
I20260807 11:17:09.339166 10265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:17:09.342357 10264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 11:17:09.349180 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.349998 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:09.366750 10068 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:09.367352 10068 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:09.365808 10117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.368266 10068 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: stopping tablet replica
I20260807 11:17:09.382819 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: NON_VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: true } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.383320 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:17:09.385098 10068 master.cc:552] Master@127.2.223.188:33277 shutdown complete.
I20260807 11:17:09.387117 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.386189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33628)
0807 11:17:09.386455 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.386876 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260807 11:17:09.387176 10265 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: attempting to promote NON_VOTER 9a820e97b0704c258ffad09d42418ca2 to VOTER
I20260807 11:17:09.387446 10068 master_runner.cc:305] Clearing existing system tablet
W20260807 11:17:09.387658 9454 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af -> Peer 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188:33277): Couldn't send request to peer 8b65b866775e4cdd81946944a14db9e8. Status: Network error: Client connection negotiation failed: client connection to 127.2.223.188:33277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:17:09.388178 10068 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:09.388314 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:09.391292 10265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } }
I20260807 11:17:09.394802 10117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 LEARNER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 11:17:09.395190 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.394696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33640)
0807 11:17:09.394843 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.394925 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 11:17:09.396020 10169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 11:17:09.399277 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.398844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33650)
0807 11:17:09.398984 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.399062 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 11:17:09.399595 9454 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af -> Peer 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188:33277): Couldn't send request to peer 8b65b866775e4cdd81946944a14db9e8. Status: Network error: Client connection negotiation failed: client connection to 127.2.223.188:33277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:17:09.399361 10169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.402161 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.402406 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:09.402324 10264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.402590 10264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:09.403012 10068 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.006s
I20260807 11:17:09.403162 10117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.408622 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.408965 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:09.410902 10271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:09.411398 10068 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:09.411721 10068 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20260807 11:17:09.411888 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "8b65b866775e4cdd81946944a14db9e8"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "dd79ea6f07542106fb29ba90785fa0a0"
server_key_iv: "301fead450684377ec2c06feffdc8de3"
server_key_version: "encryptionkey@0"
I20260807 11:17:09.415871 10068 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:09.421804 10068 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 11:17:09.421975 10068 log.cc:1196] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 11:17:09.422442 10068 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Deleting consensus metadata
I20260807 11:17:09.422732 10068 master_runner.cc:315] Copying system tablet from 127.2.223.190:40979
I20260807 11:17:09.425902 10068 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.223.190:40979
I20260807 11:17:09.427500 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9533
I20260807 11:17:09.448429 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.426801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.2.223.190:40979)
0807 11:17:09.427034 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.427039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.428300 (+ 1261us) server_negotiation.cc:407] Connection header received
0807 11:17:09.428587 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.428596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.428687 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.428783 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:09.430090 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.431181 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.432780 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.433346 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.433485 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.440802 (+ 7317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:09.440811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:09.440818 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:09.440862 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:09.444952 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.445796 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.445804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.445810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.445970 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:09.446504 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:09.446513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:09.446520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:09.446910 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:09.447048 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:09.447700 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 11:17:09.447946 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260807 11:17:09.449930 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.426806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44178)
0807 11:17:09.428174 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.428201 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.428382 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.429020 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.429027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.429631 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:09.429916 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.429930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.431418 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.431422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.432630 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.432642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.433499 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.433505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:09.433841 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:09.438840 (+ 4999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:09.438870 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:09.440478 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:09.445108 (+ 4630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.445116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.445123 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.445659 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.446120 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:09.446138 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:09.446144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:09.446331 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:09.447184 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:09.447191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:09.447397 (+ 206us) client_negotiation.cc:769] Sending connection context
0807 11:17:09.448972 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0807 11:17:09.449366 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":8822,"threads_started":1}
I20260807 11:17:09.450906 9491 tablet_copy_service.cc:140] P 7eb9c2844610439d943d7fd409bc01af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8b65b866775e4cdd81946944a14db9e8 ({username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:44178)
I20260807 11:17:09.451100 9491 tablet_copy_service.cc:161] P 7eb9c2844610439d943d7fd409bc01af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8b65b866775e4cdd81946944a14db9e8 at {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:44178: session id = 8b65b866775e4cdd81946944a14db9e8-00000000000000000000000000000000
I20260807 11:17:09.453394 9491 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:17:09.455379 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 11:17:09.459085 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:37595
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:40979
--webserver_port=35955
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:17:09.461995 10068 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: tablet copy: Starting download of 0 data blocks...
I20260807 11:17:09.462288 10068 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: tablet copy: Starting download of 1 WAL segments...
I20260807 11:17:09.464704 10068 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:17:09.467971 9491 tablet_copy_service.cc:342] P 7eb9c2844610439d943d7fd409bc01af: Request end of tablet copy session 8b65b866775e4cdd81946944a14db9e8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:44178
I20260807 11:17:09.468111 9491 tablet_copy_service.cc:433] P 7eb9c2844610439d943d7fd409bc01af: ending tablet copy session 8b65b866775e4cdd81946944a14db9e8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8b65b866775e4cdd81946944a14db9e8
I20260807 11:17:09.478928 10068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:09.486806 10279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:09.488077 10280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:09.498840 10068 server_base.cc:1034] running on GCE node
W20260807 11:17:09.499603 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:09.500094 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:09.500761 10068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:09.501964 10068 hybrid_clock.cc:648] HybridClock initialized: now 1786101429501905 us; error 47 us; skew 500 ppm
I20260807 11:17:09.502224 10068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:09 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101429, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:09.504295 10068 init.cc:377] Logged in from keytab as oryx/127.2.223.188@KRBTEST.COM (short username oryx)
I20260807 11:17:09.505587 10068 webserver.cc:460] Webserver started at http://127.2.223.188:44045/ using document root <none> and password file <none>
I20260807 11:17:09.505900 10068 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:09.506035 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:09.508864 10068 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260807 11:17:09.510628 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:09.511114 10068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 11:17:09.511262 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "8b65b866775e4cdd81946944a14db9e8"
format_stamp: "Formatted at 2026-08-07 11:17:08 on dist-test-slave-5l3k"
server_key: "dd79ea6f07542106fb29ba90785fa0a0"
server_key_iv: "301fead450684377ec2c06feffdc8de3"
server_key_version: "encryptionkey@0"
I20260807 11:17:09.511463 10068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:09.521777 10068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:09.525300 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:09.567734 10068 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.188:33277
I20260807 11:17:09.567776 10343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.188:33277 every 8 connection(s)
I20260807 11:17:09.568830 10068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
I20260807 11:17:09.582105 10344 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:17:09.586144 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Bootstrap starting.
I20260807 11:17:09.599821 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:17:09.600276 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Bootstrap complete.
I20260807 11:17:09.601133 10344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } }
I20260807 11:17:09.601415 10344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b65b866775e4cdd81946944a14db9e8, State: Initialized, Role: LEARNER
I20260807 11:17:09.601718 10344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } }
I20260807 11:17:09.602563 10348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:09.602878 10344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:09.603055 10348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:17:09.612944 10359 catalog_manager.cc:1259] Loaded cluster ID: 1c6f8d012925476e84ca5d854f65e06d
I20260807 11:17:09.613104 10359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: loading cluster ID for follower catalog manager: success
I20260807 11:17:09.616663 10359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: acquiring CA information for follower catalog manager: success
I20260807 11:17:09.619124 10359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 11:17:09.976236 10278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:09.976625 10278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:09.976872 10278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:09.986451 10278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:09.986732 10278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:09.986951 10278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:10.001158 10278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:37595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=35955
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:10.002804 10278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:10.003545 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.995776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.2.223.188:33277)
0807 11:17:09.996436 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:09.996442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:09.997092 (+ 650us) server_negotiation.cc:407] Connection header received
0807 11:17:09.997201 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:09.997208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:09.997324 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:09.997486 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:09.997942 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.999670 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.001394 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.002252 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.002409 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.002673 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.002951 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.003138 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":346,"threads_started":1}
I20260807 11:17:10.003659 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:09.995408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33656)
0807 11:17:09.996917 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:09.996944 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:09.997078 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:09.997493 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:09.997496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:09.997517 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:09.997789 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:09.997798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:09.999819 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:09.999823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.001197 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.001210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.002471 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.002494 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.002758 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.002785 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.002923 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.003140 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"thread_start_us":1261,"threads_started":1}
I20260807 11:17:10.004863 10278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 11:17:10.005105 10318 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 11:17:10.006006 10348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb9c2844610439d943d7fd409bc01af. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:10.006423 10348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: LEARNER
W20260807 11:17:10.029862 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:10.029911 10366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:10.030650 10369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:10.032379 10278 server_base.cc:1034] running on GCE node
I20260807 11:17:10.033361 10278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:10.034893 10278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:10.036162 10278 hybrid_clock.cc:648] HybridClock initialized: now 1786101430036109 us; error 85 us; skew 500 ppm
I20260807 11:17:10.036589 10278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:10.042773 10278 init.cc:377] Logged in from keytab as oryx/127.2.223.129@KRBTEST.COM (short username oryx)
I20260807 11:17:10.045363 10278 webserver.cc:460] Webserver started at http://127.2.223.129:35955/ using document root <none> and password file <none>
I20260807 11:17:10.046712 10278 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:10.047004 10278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:10.055308 10278 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:10.059975 10377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.061909 10278 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:10.062156 10278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "2c2dae56828b4222b2f13b759e771dfd"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "4bd814cdb35fb46e7de7777fdd02fd6a"
server_key_iv: "1e931a1900d705b9bd7c08acdddaaffd"
server_key_version: "encryptionkey@0"
I20260807 11:17:10.063594 10278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:10.105449 10278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:10.109879 10278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:10.110139 10278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:10.111220 10278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:10.112733 10278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:10.112848 10278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.112998 10278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:10.113162 10278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.189380 10278 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:37595
I20260807 11:17:10.189531 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:37595 every 8 connection(s)
I20260807 11:17:10.191128 10278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:10.199342 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 10278
I20260807 11:17:10.200065 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9673
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.130@KRBTEST.COM
I20260807 11:17:10.243830 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.197305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:41145 (local address 127.2.223.188:33277)
0807 11:17:10.197575 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.197583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.203683 (+ 6100us) server_negotiation.cc:407] Connection header received
0807 11:17:10.209809 (+ 6126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.209818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.209933 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.210024 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.214600 (+ 4576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.216244 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.219570 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.220444 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.220653 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.233052 (+ 12399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.233061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.233069 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.233119 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.238191 (+ 5072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.239714 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.239721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.239727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.239951 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.240705 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.240713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.240717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.241199 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.241357 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.242736 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.243123 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:17:10.246837 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.196318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.2.223.129:41145)
0807 11:17:10.203378 (+ 7060us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.203502 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.209604 (+ 6102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.210752 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.210770 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.212389 (+ 1619us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.214313 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.214345 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.216607 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.216613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.219391 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.219405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.222153 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.222163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.222768 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.226093 (+ 3325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.226134 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.232140 (+ 6006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.238416 (+ 6276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.238428 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.238450 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.239559 (+ 1109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.240122 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.240129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.240134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.240528 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.241491 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.241510 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.242100 (+ 590us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.244232 (+ 2132us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.244864 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6527,"spinlock_wait_cycles":9472,"thread_start_us":6193,"threads_started":1}
I20260807 11:17:10.248335 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.197485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:38961)
0807 11:17:10.204448 (+ 6963us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.204471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.208990 (+ 4519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.219599 (+ 10609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.219604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.220228 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.220788 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.220803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.223433 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.223440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.225673 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.225690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.227506 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.227529 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.230284 (+ 2755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.232565 (+ 2281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.232596 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.236877 (+ 4281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.243852 (+ 6975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.243859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.243863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.244409 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.244956 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.244964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.244970 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.245252 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.246606 (+ 1354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.246615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.246881 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.247438 (+ 557us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.247828 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6760,"mutex_wait_us":2384,"thread_start_us":6403,"threads_started":1}
I20260807 11:17:10.248970 10495 heartbeater.cc:347] Connected to a master server at 127.2.223.188:33277
I20260807 11:17:10.249378 10495 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.249756 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.195273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38961 (local address 127.2.223.190:40979)
0807 11:17:10.196078 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.196084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.206697 (+ 10613us) server_negotiation.cc:407] Connection header received
0807 11:17:10.209426 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.209436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.209556 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.209695 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.221024 (+ 11329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.222706 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.225864 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.226887 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.227050 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.237251 (+ 10201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.237259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.237264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.237311 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.243706 (+ 6395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.244577 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.244586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.244592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.244822 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.245401 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.245407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.245410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.245973 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.246170 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.248733 (+ 2563us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.249013 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":493,"threads_started":1}
I20260807 11:17:10.250949 10495 heartbeater.cc:511] Master 127.2.223.188:33277 requested a full tablet report, sending...
I20260807 11:17:10.252218 10496 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:10.252424 10496 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.253063 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.201708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.2.223.130:33053)
0807 11:17:10.214708 (+ 13000us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.214725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.217362 (+ 2637us) server_negotiation.cc:407] Connection header received
0807 11:17:10.217503 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.217514 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.217848 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.218048 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.223913 (+ 5865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.228137 (+ 4224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.231720 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.232872 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.233052 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.241744 (+ 8692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.241759 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.241778 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.241834 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.247486 (+ 5652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.248696 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.248705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.248709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.249033 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.249525 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.249531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.249535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.249945 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.251831 (+ 1886us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.252022 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.252333 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12725,"thread_start_us":14019,"threads_started":1}
I20260807 11:17:10.253244 10496 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:10.253485 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.208224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.2.223.129:42021)
0807 11:17:10.215440 (+ 7216us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.215488 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.215664 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.221728 (+ 6064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.221733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.223132 (+ 1399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.224472 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.224492 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.226258 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.226263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.229378 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.229394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.231667 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.231675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.231883 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.232752 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.232773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.237744 (+ 4971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.245823 (+ 8079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.245828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.245832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.246438 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.250548 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.250554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.250558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.250779 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.251631 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.251636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.251893 (+ 257us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.252317 (+ 424us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.252692 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7028,"mutex_wait_us":1,"thread_start_us":6808,"threads_started":1}
I20260807 11:17:10.254463 10308 ts_manager.cc:194] Registered new tserver with Master: 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:10.254642 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.204699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:42021 (local address 127.2.223.189:37579)
0807 11:17:10.221412 (+ 16713us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.221417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.221434 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:17:10.221483 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.221487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.221558 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.221699 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.224702 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.226018 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.229621 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.230725 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.231251 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.238098 (+ 6847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.238106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.238110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.238156 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.245625 (+ 7469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.250035 (+ 4410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.250054 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.250057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.250301 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.250953 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.250959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.250963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.251428 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.251599 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.253852 (+ 2253us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.254089 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16604,"thread_start_us":16495,"threads_started":1}
I20260807 11:17:10.254940 9805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:10.255999 9470 ts_manager.cc:194] Re-registered known tserver with Master: 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:10.256840 10497 heartbeater.cc:347] Connected to a master server at 127.2.223.189:37579
I20260807 11:17:10.256923 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:38961
I20260807 11:17:10.257076 10497 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.257944 10497 heartbeater.cc:511] Master 127.2.223.189:37579 requested a full tablet report, sending...
I20260807 11:17:10.263386 10106 ts_manager.cc:194] Registered new tserver with Master: 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:10.343156 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9673
I20260807 11:17:10.365265 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:33053
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:40979
--webserver_port=39377
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:17:10.450367 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: NON_VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: true } } }
I20260807 11:17:10.450803 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:10.452764 10169 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: attempting to promote NON_VOTER 8b65b866775e4cdd81946944a14db9e8 to VOTER
I20260807 11:17:10.453307 10169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } }
I20260807 11:17:10.456107 10318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 LEARNER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 11:17:10.456238 10117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Refusing update from remote peer 7eb9c2844610439d943d7fd409bc01af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 11:17:10.457054 10169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:17:10.457422 10509 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:17:10.460713 10169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.462658 10318 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.464555 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.464794 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:10.465644 10117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.466449 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.466671 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:10.465173 10509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.466822 10509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:10.467713 10348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.468029 10348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:10.470659 10515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb9c2844610439d943d7fd409bc01af" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:10.471040 10515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 11:17:10.728223 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:10.728494 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:10.728696 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:10.736057 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:10.736158 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:10.736357 10507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:10.746315 10507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:33053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=39377
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:10.747664 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:10.749104 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:10.762907 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:10.763015 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:10.762943 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:10.763928 10507 server_base.cc:1034] running on GCE node
I20260807 11:17:10.764544 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:10.765453 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:10.766683 10507 hybrid_clock.cc:648] HybridClock initialized: now 1786101430766644 us; error 52 us; skew 500 ppm
I20260807 11:17:10.766985 10507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:10.770187 10507 init.cc:377] Logged in from keytab as oryx/127.2.223.130@KRBTEST.COM (short username oryx)
I20260807 11:17:10.771723 10507 webserver.cc:460] Webserver started at http://127.2.223.130:39377/ using document root <none> and password file <none>
I20260807 11:17:10.772356 10507 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:10.772469 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:10.777627 10507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:17:10.780265 10532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.781558 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:17:10.781694 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "defc27725dbf4ab08e4f5646b5bc1de6"
format_stamp: "Formatted at 2026-08-07 11:17:06 on dist-test-slave-5l3k"
server_key: "d6f9184d0453fcf4a18fbb27adc39a7b"
server_key_iv: "71f3909f3d85ae200697042f7b43a80c"
server_key_version: "encryptionkey@0"
I20260807 11:17:10.782749 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:10.816713 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:10.822705 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:10.823050 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:10.824241 10507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:10.826097 10507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:10.826210 10507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.826300 10507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:10.826397 10507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:10.868599 10507 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:33053
I20260807 11:17:10.868772 10650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:33053 every 8 connection(s)
I20260807 11:17:10.870393 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:10.872349 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 10507
I20260807 11:17:10.872648 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9816
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.131@KRBTEST.COM
Aug 07 11:17:10 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101430, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:10.915069 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.875812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:50023 (local address 127.2.223.189:37579)
0807 11:17:10.882265 (+ 6453us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.882271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.882292 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:17:10.882366 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.882375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.882525 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.882667 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.889230 (+ 6563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.890976 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.895056 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.895992 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.896112 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.903812 (+ 7700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.903820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.903828 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.903927 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.909230 (+ 5303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.910639 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.910647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.910651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.910944 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.912012 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.912017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.912020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.912631 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.912791 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.914066 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.914380 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6322,"thread_start_us":6189,"threads_started":1}
I20260807 11:17:10.917414 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.879626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.2.223.130:40711)
0807 11:17:10.886300 (+ 6674us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.886317 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.886422 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.889273 (+ 2851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.889278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.890011 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.891891 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.891925 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.894523 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.894529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.897536 (+ 3007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.897556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.899387 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.899397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.899665 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.900557 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.900581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.904740 (+ 4159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.911261 (+ 6521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.911269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.911274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.912389 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.912992 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.912997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.913012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.913292 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.914113 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.914117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.914313 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.914523 (+ 210us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.915129 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6549,"mutex_wait_us":39,"thread_start_us":9244,"threads_started":1}
I20260807 11:17:10.917706 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.880210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:40711 (local address 127.2.223.188:33277)
0807 11:17:10.888139 (+ 7929us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.888148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.888173 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:17:10.888252 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.888260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.888361 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.891088 (+ 2727us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.892166 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.894201 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.897786 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.898850 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.898983 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.904971 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.904996 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.905004 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.905061 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.910971 (+ 5910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.912563 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.912572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.912577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.912827 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.913458 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.913465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.913469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.913873 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.914033 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.916890 (+ 2857us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.917159 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7758,"thread_start_us":7647,"threads_started":1}
I20260807 11:17:10.918759 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.878389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:42027 (local address 127.2.223.190:40979)
0807 11:17:10.883326 (+ 4937us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.883335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.889483 (+ 6148us) server_negotiation.cc:407] Connection header received
0807 11:17:10.889829 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.889835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.889976 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.890271 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.892735 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.894167 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.898952 (+ 4785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.900122 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.900228 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.908298 (+ 8070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.908306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.908313 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.908421 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.914598 (+ 6177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.915789 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.915796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.915800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.916022 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.916561 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.916566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.916570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.917043 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.917188 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.918039 (+ 851us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.918272 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4765,"thread_start_us":4651,"threads_started":1}
I20260807 11:17:10.917361 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.874984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.2.223.130:50023)
0807 11:17:10.877572 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.877630 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.880897 (+ 3267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.886892 (+ 5995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.886904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.887796 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.888863 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.888904 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.891256 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.891264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.894836 (+ 3572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.894851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.896661 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.896670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.897394 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.898621 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.898655 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.903503 (+ 4848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.909408 (+ 5905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.909421 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.909441 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.910455 (+ 1014us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.911217 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.911222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.911246 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.911852 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.912928 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.912936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.913527 (+ 591us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.915681 (+ 2154us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.916000 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2256,"mutex_wait_us":199,"thread_start_us":1975,"threads_started":1}
I20260807 11:17:10.921175 10651 heartbeater.cc:347] Connected to a master server at 127.2.223.188:33277
I20260807 11:17:10.918794 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.882260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:42027)
0807 11:17:10.889339 (+ 7079us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:10.889399 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:10.889639 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:10.890616 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:10.890621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:10.891571 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:10.892509 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.892528 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.894363 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.894368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.898737 (+ 4369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:10.898760 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.900738 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.900748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:10.900992 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.903770 (+ 2778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:10.903800 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:10.907977 (+ 4177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:10.915052 (+ 7075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.915066 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.915069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.915656 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.916160 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:10.916165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:10.916169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:10.916428 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:10.917172 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:10.917177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:10.917719 (+ 542us) client_negotiation.cc:769] Sending connection context
0807 11:17:10.917916 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 11:17:10.918177 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6905,"mutex_wait_us":51,"thread_start_us":3677,"threads_started":1}
I20260807 11:17:10.921546 10651 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.922580 10653 heartbeater.cc:347] Connected to a master server at 127.2.223.189:37579
I20260807 11:17:10.922747 10652 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:10.922780 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:10.873853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.2.223.131:37973)
0807 11:17:10.886408 (+ 12555us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:10.886419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:10.887085 (+ 666us) server_negotiation.cc:407] Connection header received
0807 11:17:10.887576 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:10.887586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:10.887869 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:10.888110 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:10.892455 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.894872 (+ 2417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.897159 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:10.898152 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:10.898323 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:10.908484 (+ 10161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:10.908504 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:10.908524 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:10.908597 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:10.914788 (+ 6191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.916288 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.916297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.916300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.917334 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:10.918694 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:10.918701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:10.918704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:10.919257 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:10.919758 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:10.920302 (+ 544us) server_negotiation.cc:299] Negotiation successful
0807 11:17:10.920725 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12322,"thread_start_us":13357,"threads_started":1}
I20260807 11:17:10.922832 10653 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.922943 10652 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:10.924119 10651 heartbeater.cc:511] Master 127.2.223.188:33277 requested a full tablet report, sending...
I20260807 11:17:10.924441 9947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:10.924499 10652 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:10.924515 10653 heartbeater.cc:511] Master 127.2.223.189:37579 requested a full tablet report, sending...
I20260807 11:17:10.926365 10106 ts_manager.cc:194] Registered new tserver with Master: defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053)
I20260807 11:17:10.926642 10308 ts_manager.cc:194] Registered new tserver with Master: defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053)
I20260807 11:17:10.927340 9470 ts_manager.cc:194] Re-registered known tserver with Master: defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053)
I20260807 11:17:10.928349 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:42027
I20260807 11:17:11.016867 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9816
I20260807 11:17:11.038564 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:37973
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:40979
--webserver_port=42449
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:17:11.262008 10496 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
W20260807 11:17:11.382850 10664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:11.383170 10664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:11.383337 10664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:11.390849 10664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:11.391010 10664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:11.391194 10664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:11.400890 10664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:37973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=42449
--tserver_master_addrs=127.2.223.190:40979,127.2.223.189:37579,127.2.223.188:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:11.402261 10664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:11.403864 10664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:11.418197 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:11.418929 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:11.418973 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:11.419641 10664 server_base.cc:1034] running on GCE node
I20260807 11:17:11.420389 10664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:11.421303 10664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:11.422585 10664 hybrid_clock.cc:648] HybridClock initialized: now 1786101431422517 us; error 44 us; skew 500 ppm
I20260807 11:17:11.422907 10664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:11 dist-test-slave-5l3k krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101431, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:11.426085 10664 init.cc:377] Logged in from keytab as oryx/127.2.223.131@KRBTEST.COM (short username oryx)
I20260807 11:17:11.427625 10664 webserver.cc:460] Webserver started at http://127.2.223.131:42449/ using document root <none> and password file <none>
I20260807 11:17:11.428375 10664 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:11.428491 10664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:11.433719 10664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:17:11.436342 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:11.437640 10664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:17:11.437780 10664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "495ff7334ca549d9a4a788d2071cb08f"
format_stamp: "Formatted at 2026-08-07 11:17:07 on dist-test-slave-5l3k"
server_key: "15b07f5cfec2b99e0a57b9448cc2265f"
server_key_iv: "350e935c56906c2c4285f84aabb56ac7"
server_key_version: "encryptionkey@0"
I20260807 11:17:11.438783 10664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:11.475621 10664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:11.479655 10664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:11.479885 10664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:11.480849 10664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:11.482218 10664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:11.482306 10664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:11.482390 10664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:11.482446 10664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:11.518692 10664 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37973
I20260807 11:17:11.518781 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37973 every 8 connection(s)
I20260807 11:17:11.520179 10664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:11.525481 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 10664
Aug 07 11:17:11 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101431, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
Aug 07 11:17:11 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101431, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.188@KRBTEST.COM
I20260807 11:17:11.571718 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.523242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:33759 (local address 127.2.223.190:40979)
0807 11:17:11.541101 (+ 17859us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.541107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.544104 (+ 2997us) server_negotiation.cc:407] Connection header received
0807 11:17:11.545621 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.545628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.545726 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.545852 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.548798 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.550262 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.554640 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.555533 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.555690 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.562824 (+ 7134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.562831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.562835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.562876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.566903 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.568278 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.568287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.568292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.568441 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.569429 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.569436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.569440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.569839 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.569995 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.571092 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.571334 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17686,"thread_start_us":17855,"threads_started":1}
Aug 07 11:17:11 dist-test-slave-5l3k krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101431, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.189@KRBTEST.COM
I20260807 11:17:11.576479 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.523266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:58155 (local address 127.2.223.188:33277)
0807 11:17:11.542066 (+ 18800us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.542074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.542101 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:11.546131 (+ 4030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.546139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.546243 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.546360 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.549919 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.551524 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.559374 (+ 7850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.560075 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.560175 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.566724 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.566731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.566735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.566782 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.570722 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.572367 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.572375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.572380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.572579 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.574215 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.574221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.574224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.574618 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.574768 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.575763 (+ 995us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.575997 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18558,"thread_start_us":18462,"threads_started":1}
I20260807 11:17:11.578364 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.523987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.2.223.131:58155)
0807 11:17:11.535702 (+ 11715us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.535775 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.545284 (+ 9509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.546667 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.546674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.549371 (+ 2697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:11.549704 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.549717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.556356 (+ 6639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.556363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.559188 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.559201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.560476 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.560483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.560770 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.561704 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:11.561728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:11.564390 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:11.570877 (+ 6487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.570885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.570891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.571364 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.572763 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.572767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.572771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.572962 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.574886 (+ 1924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:11.574892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:11.575056 (+ 164us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.576832 (+ 1776us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.577140 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11310,"mutex_wait_us":1195,"thread_start_us":20043,"threads_started":1}
I20260807 11:17:11.578398 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.523987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:33759)
0807 11:17:11.543964 (+ 19977us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.543990 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.545302 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.546126 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.546144 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.547502 (+ 1358us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:11.548575 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.548605 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.551752 (+ 3147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.551760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.554450 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.554470 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.556981 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.556989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.557266 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.559198 (+ 1932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:11.559239 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:11.562577 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:11.567137 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.567149 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.567167 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.568133 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.568607 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.568612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.568617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.569282 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.570110 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:11.570118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:11.570596 (+ 478us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.572896 (+ 2300us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.573526 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19759,"mutex_wait_us":43,"spinlock_wait_cycles":54656,"thread_start_us":14338,"threads_started":1}
I20260807 11:17:11.580335 10800 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40979
I20260807 11:17:11.580706 10800 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:11.581825 10800 heartbeater.cc:511] Master 127.2.223.190:40979 requested a full tablet report, sending...
I20260807 11:17:11.582686 10799 heartbeater.cc:347] Connected to a master server at 127.2.223.188:33277
I20260807 11:17:11.582871 10799 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:11.583154 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.540006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:49231 (local address 127.2.223.189:37579)
0807 11:17:11.543849 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.543858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.544582 (+ 724us) server_negotiation.cc:407] Connection header received
0807 11:17:11.545714 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.545719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.548819 (+ 3100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.548947 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.554706 (+ 5759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.556132 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.558010 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.558695 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.558821 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.574193 (+ 15372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.574199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.574204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.574247 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.577950 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.579525 (+ 1575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.579534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.579540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.580466 (+ 926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.581144 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.581152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.581158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.581539 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.581682 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.582389 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.582653 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3409,"thread_start_us":3272,"threads_started":1}
I20260807 11:17:11.583415 10799 heartbeater.cc:511] Master 127.2.223.188:33277 requested a full tablet report, sending...
I20260807 11:17:11.583523 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.542717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.2.223.131:49231)
0807 11:17:11.544453 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.544471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.545302 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.552096 (+ 6794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.552101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.552738 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:11.553079 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.553092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.556268 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.556277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.557846 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.557859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.561123 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.561129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.561329 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.564717 (+ 3388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:11.564741 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:11.573971 (+ 9230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:11.578906 (+ 4935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.578911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.578914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.579403 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.580648 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:11.580653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:11.580656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:11.580878 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:11.581984 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:11.581989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:11.582120 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.582275 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.582594 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":42,"thread_start_us":913,"threads_started":1}
I20260807 11:17:11.584798 10801 heartbeater.cc:347] Connected to a master server at 127.2.223.189:37579
I20260807 11:17:11.584959 10801 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:11.585618 10801 heartbeater.cc:511] Master 127.2.223.189:37579 requested a full tablet report, sending...
I20260807 11:17:11.586469 9470 ts_manager.cc:194] Re-registered known tserver with Master: 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:11.586918 10308 ts_manager.cc:194] Registered new tserver with Master: 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:11.587754 9470 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:33759
I20260807 11:17:11.589349 10106 ts_manager.cc:194] Registered new tserver with Master: 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:11.607193 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.545937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.2.223.190:40979)
0807 11:17:11.572831 (+ 26894us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.572840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.580526 (+ 7686us) server_negotiation.cc:407] Connection header received
0807 11:17:11.580750 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.580756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.580849 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.580943 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.588117 (+ 7174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.589225 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.591774 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.592667 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.592783 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.598358 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.598363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.598367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.598397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.603033 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.604355 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.604360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.604362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.604497 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.605015 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.605020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.605023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.605460 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.605864 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.606375 (+ 511us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.606691 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26029,"thread_start_us":39313,"threads_started":1}
I20260807 11:17:11.618441 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.547931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.2.223.188:33277)
0807 11:17:11.576755 (+ 28824us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.576763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.589498 (+ 12735us) server_negotiation.cc:407] Connection header received
0807 11:17:11.592457 (+ 2959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.592466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.592631 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.592756 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.597488 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.598988 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.600846 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.601611 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.601761 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.607259 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.607266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.607270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.607310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.613840 (+ 6530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.615063 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.615070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.615074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.615363 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.615893 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.615898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.615902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.616289 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.616681 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.617512 (+ 831us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.617830 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28711,"thread_start_us":45055,"threads_started":1}
I20260807 11:17:11.618525 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.564059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.2.223.189:37579)
0807 11:17:11.579851 (+ 15792us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.579857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.592500 (+ 12643us) server_negotiation.cc:407] Connection header received
0807 11:17:11.592783 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.592790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.592892 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.592997 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:11.597457 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.598982 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.601561 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.602404 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.602559 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.608146 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:11.608157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:11.608161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:11.608200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:11.613667 (+ 5467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.614720 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.614727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.614732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.614995 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:11.615608 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:11.615613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:11.615616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:11.616044 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:11.616335 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.617413 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.617686 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15680,"thread_start_us":15782,"threads_started":1}
I20260807 11:17:11.613901 9470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:11.626200 9470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:11.679730 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.653480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.2.223.129:37595)
0807 11:17:11.654453 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.654467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.660926 (+ 6459us) server_negotiation.cc:407] Connection header received
0807 11:17:11.661142 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.661154 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.661489 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.661798 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:11.664816 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.667418 (+ 2602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.673654 (+ 6236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.675579 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.675714 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.675942 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.676694 (+ 752us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.679220 (+ 2526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":599,"threads_started":1}
I20260807 11:17:11.681942 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.653148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37595 (local address 127.0.0.1:49580)
0807 11:17:11.659127 (+ 5979us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.659151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.659377 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.663195 (+ 3818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.663204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.663242 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:11.664011 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.664030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.668244 (+ 4214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.668253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.673453 (+ 5200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.673468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.675764 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.675769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.676007 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.676063 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.676180 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.676407 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5724,"mutex_wait_us":1,"thread_start_us":798,"threads_started":1}
I20260807 11:17:11.686846 10426 tablet_service.cc:1467] Processing CreateTablet for tablet 51849447f8b843e293e898c86afc8866 (DEFAULT_TABLE table=test-table [id=6bfbe0d901fa4927a717d814ecdd33b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:11.687778 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.652546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:33053 (local address 127.0.0.1:59936)
0807 11:17:11.654840 (+ 2294us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.654865 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.666884 (+ 12019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.670180 (+ 3296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.670188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.670229 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:11.671700 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.671722 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.675584 (+ 3862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.675590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.683044 (+ 7454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.683062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.686102 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.686108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.686658 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.686721 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.686914 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.687069 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"thread_start_us":683,"threads_started":1}
I20260807 11:17:11.688901 10426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51849447f8b843e293e898c86afc8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:11.688943 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.652752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.2.223.130:33053)
0807 11:17:11.669432 (+ 16680us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.669442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.669476 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:17:11.669600 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.669609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.669927 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.670143 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:11.672070 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.675359 (+ 3289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.683413 (+ 8054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.685858 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.686056 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.686324 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.688284 (+ 1960us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.688537 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16450,"thread_start_us":17663,"threads_started":1}
I20260807 11:17:11.693253 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.654659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:37973 (local address 127.0.0.1:46524)
0807 11:17:11.680946 (+ 26287us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.680968 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.681179 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.681780 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.681784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.681813 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:11.682610 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.682626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.686778 (+ 4152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.686785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.689108 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.689125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.692045 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.692054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.692360 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.692413 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.692552 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.692713 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26141,"thread_start_us":25762,"threads_started":1}
I20260807 11:17:11.693365 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.655085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.2.223.131:37973)
0807 11:17:11.679715 (+ 24630us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.679727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.681068 (+ 1341us) server_negotiation.cc:407] Connection header received
0807 11:17:11.681359 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.681366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.681593 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.681817 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:11.683044 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.686651 (+ 3607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.689305 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.691647 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.691815 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.692043 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.692622 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.692827 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24360,"thread_start_us":24643,"threads_started":1}
I20260807 11:17:11.693092 10583 tablet_service.cc:1467] Processing CreateTablet for tablet 51849447f8b843e293e898c86afc8866 (DEFAULT_TABLE table=test-table [id=6bfbe0d901fa4927a717d814ecdd33b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:11.695286 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51849447f8b843e293e898c86afc8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:11.698118 10731 tablet_service.cc:1467] Processing CreateTablet for tablet 51849447f8b843e293e898c86afc8866 (DEFAULT_TABLE table=test-table [id=6bfbe0d901fa4927a717d814ecdd33b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:11.699863 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51849447f8b843e293e898c86afc8866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:11.710091 10836 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Bootstrap starting.
I20260807 11:17:11.715482 10837 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Bootstrap starting.
I20260807 11:17:11.716922 10836 tablet_bootstrap.cc:654] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:11.718427 10836 log.cc:824] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:11.720131 10838 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Bootstrap starting.
I20260807 11:17:11.720449 10837 tablet_bootstrap.cc:654] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:11.723039 10837 log.cc:824] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:11.726111 10838 tablet_bootstrap.cc:654] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:11.727221 10836 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: No bootstrap required, opened a new log
I20260807 11:17:11.727469 10837 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: No bootstrap required, opened a new log
I20260807 11:17:11.727670 10836 ts_tablet_manager.cc:1397] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260807 11:17:11.727852 10837 ts_tablet_manager.cc:1397] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260807 11:17:11.728605 10838 log.cc:824] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:11.733212 10838 tablet_bootstrap.cc:492] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: No bootstrap required, opened a new log
I20260807 11:17:11.733662 10838 ts_tablet_manager.cc:1397] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260807 11:17:11.739013 10836 raft_consensus.cc:348] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.739494 10836 raft_consensus.cc:374] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:11.739764 10836 raft_consensus.cc:729] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2dae56828b4222b2f13b759e771dfd, State: Initialized, Role: FOLLOWER
I20260807 11:17:11.740475 10836 consensus_queue.cc:260] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.743758 10837 raft_consensus.cc:348] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.744393 10837 raft_consensus.cc:374] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:11.744540 10837 raft_consensus.cc:729] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc27725dbf4ab08e4f5646b5bc1de6, State: Initialized, Role: FOLLOWER
I20260807 11:17:11.745299 10837 consensus_queue.cc:260] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.748813 10836 ts_tablet_manager.cc:1428] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260807 11:17:11.751034 10838 raft_consensus.cc:348] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.752333 10838 raft_consensus.cc:374] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:11.752547 10838 raft_consensus.cc:729] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ff7334ca549d9a4a788d2071cb08f, State: Initialized, Role: FOLLOWER
I20260807 11:17:11.754581 10838 consensus_queue.cc:260] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.755599 10837 ts_tablet_manager.cc:1428] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Time spent starting tablet: real 0.028s user 0.018s sys 0.007s
I20260807 11:17:11.755985 10652 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
I20260807 11:17:11.756654 10800 heartbeater.cc:503] Master 127.2.223.190:40979 was elected leader, sending a full tablet report...
I20260807 11:17:11.759351 10838 ts_tablet_manager.cc:1428] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Time spent starting tablet: real 0.026s user 0.012s sys 0.010s
W20260807 11:17:11.773248 10802 tablet.cc:2369] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:11.885962 10654 tablet.cc:2369] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:11.947135 10498 tablet.cc:2369] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:17:11.962075 10844 raft_consensus.cc:482] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:11.962630 10844 raft_consensus.cc:504] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.964864 10844 leader_election.cc:288] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595), 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:11.976974 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.966417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:55711 (local address 127.2.223.131:37973)
0807 11:17:11.966652 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.966657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.967641 (+ 984us) server_negotiation.cc:407] Connection header received
0807 11:17:11.967855 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.967859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.967926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.968008 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:11.968533 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.969841 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.973500 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.975565 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.975803 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.975877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.976527 (+ 650us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.976671 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 11:17:11.977247 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.966178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:37973 (local address 127.2.223.130:55711)
0807 11:17:11.967484 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.967509 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.967681 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.968053 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.968059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.968088 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:11.968359 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.968367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.969965 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.969969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.973252 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.973275 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.975971 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.975980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.976249 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.976352 (+ 103us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.976472 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.976683 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":231,"thread_start_us":822,"threads_started":1}
I20260807 11:17:11.977885 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.965302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:58325 (local address 127.2.223.129:37595)
0807 11:17:11.965531 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:11.965536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:11.966370 (+ 834us) server_negotiation.cc:407] Connection header received
0807 11:17:11.966708 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:11.966718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:11.966840 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:11.966974 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:11.968175 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.969739 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.973518 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.975435 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.975759 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.975851 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:11.977354 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0807 11:17:11.977509 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:17:11.977969 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:11.965391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37595 (local address 127.2.223.130:58325)
0807 11:17:11.966232 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:11.966254 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:11.966499 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:11.967061 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:11.967098 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:11.967137 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:11.967962 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.967974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.969958 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.969964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.973280 (+ 3316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:11.973290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:11.976032 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:11.976038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:11.976251 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:11.976352 (+ 101us) client_negotiation.cc:769] Sending connection context
0807 11:17:11.976472 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:17:11.976685 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":208,"thread_start_us":397,"threads_started":1}
I20260807 11:17:11.978905 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51849447f8b843e293e898c86afc8866" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ff7334ca549d9a4a788d2071cb08f" is_pre_election: true
I20260807 11:17:11.979166 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51849447f8b843e293e898c86afc8866" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2dae56828b4222b2f13b759e771dfd" is_pre_election: true
I20260807 11:17:11.979740 10753 raft_consensus.cc:2454] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 0.
I20260807 11:17:11.979777 10448 raft_consensus.cc:2454] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 0.
I20260807 11:17:11.980487 10536 leader_election.cc:302] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ff7334ca549d9a4a788d2071cb08f, defc27725dbf4ab08e4f5646b5bc1de6; no voters:
I20260807 11:17:11.980921 10844 raft_consensus.cc:2781] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:11.981186 10844 raft_consensus.cc:482] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:11.981405 10844 raft_consensus.cc:3037] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:11.984830 10844 raft_consensus.cc:504] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.985890 10844 leader_election.cc:288] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 election: Requested vote from peers 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595), 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973)
I20260807 11:17:11.986562 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51849447f8b843e293e898c86afc8866" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2dae56828b4222b2f13b759e771dfd"
I20260807 11:17:11.986876 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51849447f8b843e293e898c86afc8866" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ff7334ca549d9a4a788d2071cb08f"
I20260807 11:17:11.986901 10448 raft_consensus.cc:3037] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:11.987151 10753 raft_consensus.cc:3037] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:11.990080 10448 raft_consensus.cc:2454] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 1.
I20260807 11:17:11.990171 10753 raft_consensus.cc:2454] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 1.
I20260807 11:17:11.990718 10536 leader_election.cc:302] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2dae56828b4222b2f13b759e771dfd, defc27725dbf4ab08e4f5646b5bc1de6; no voters:
I20260807 11:17:11.991084 10844 raft_consensus.cc:2781] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:11.991976 10844 raft_consensus.cc:686] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 LEADER]: Becoming Leader. State: Replica: defc27725dbf4ab08e4f5646b5bc1de6, State: Running, Role: LEADER
I20260807 11:17:11.992784 10844 consensus_queue.cc:237] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } }
I20260807 11:17:11.999087 9467 catalog_manager.cc:5495] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } health_report { overall_health: UNKNOWN } } }
I20260807 11:17:12.065796 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:12.059320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.2.223.130:33053)
0807 11:17:12.059579 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:12.059583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:12.059603 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:17:12.059705 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:12.059709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:12.059894 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:12.060093 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:12.060584 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.061706 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.063312 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.063784 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.063948 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:12.064550 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:12.065090 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:12.065274 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:12.065364 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 11:17:12.065454 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:17:12.076272 10448 raft_consensus.cc:1261] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Refusing update from remote peer defc27725dbf4ab08e4f5646b5bc1de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:12.076387 10753 raft_consensus.cc:1261] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Refusing update from remote peer defc27725dbf4ab08e4f5646b5bc1de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:12.077507 10848 consensus_queue.cc:1035] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:12.078015 10844 consensus_queue.cc:1035] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:12.096295 10853 mvcc.cc:205] Tried to move back new op lower bound from 7315871465773457408 to 7315871465446830080. Current Snapshot: MvccSnapshot[applied={T|T < 7315871465773457408}]
I20260807 11:17:12.117121 9467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:12.117398 9467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:12.125087 9467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb9c2844610439d943d7fd409bc01af: Sending DeleteTablet for 3 replicas of tablet 51849447f8b843e293e898c86afc8866
I20260807 11:17:12.126359 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet 51849447f8b843e293e898c86afc8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:12 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:59936
I20260807 11:17:12.126756 10426 tablet_service.cc:1514] Processing DeleteTablet for tablet 51849447f8b843e293e898c86afc8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:12 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:49580
I20260807 11:17:12.126844 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet 51849447f8b843e293e898c86afc8866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:12 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:46524
I20260807 11:17:12.139647 10862 tablet_replica.cc:330] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: stopping tablet replica
I20260807 11:17:12.141016 10862 raft_consensus.cc:2229] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:12.142829 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9436
I20260807 11:17:12.144532 9502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:12.153724 10863 tablet_replica.cc:330] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: stopping tablet replica
I20260807 11:17:12.143406 10862 raft_consensus.cc:2258] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:12.155198 10863 raft_consensus.cc:2229] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:12.156002 10863 raft_consensus.cc:2258] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:12.160465 10862 ts_tablet_manager.cc:1905] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:12.160571 10863 ts_tablet_manager.cc:1905] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:12.161180 10864 tablet_replica.cc:330] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: stopping tablet replica
I20260807 11:17:12.163322 10864 raft_consensus.cc:2229] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:12.164233 10864 raft_consensus.cc:2258] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:12.171378 10863 ts_tablet_manager.cc:1918] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:12.171705 10863 log.cc:1196] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/51849447f8b843e293e898c86afc8866
I20260807 11:17:12.171854 10862 ts_tablet_manager.cc:1918] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:12.172241 10862 log.cc:1196] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/51849447f8b843e293e898c86afc8866
I20260807 11:17:12.172919 10863 ts_tablet_manager.cc:1939] T 51849447f8b843e293e898c86afc8866 P 495ff7334ca549d9a4a788d2071cb08f: Deleting consensus metadata
I20260807 11:17:12.173269 10862 ts_tablet_manager.cc:1939] T 51849447f8b843e293e898c86afc8866 P defc27725dbf4ab08e4f5646b5bc1de6: Deleting consensus metadata
I20260807 11:17:12.175354 10864 ts_tablet_manager.cc:1905] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:12.182174 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:12.130112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.2.223.189:37579)
0807 11:17:12.139245 (+ 9133us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:12.139250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:12.142615 (+ 3365us) server_negotiation.cc:407] Connection header received
0807 11:17:12.148571 (+ 5956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:12.148580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:12.148689 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:12.148814 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:12.157044 (+ 8230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.158844 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.165951 (+ 7107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.166738 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.166866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:12.171872 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:12.171880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:12.171885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:12.171929 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:12.177210 (+ 5281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.178248 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.178254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.178257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.178435 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.179038 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.179043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.179047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.179445 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:12.179763 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:12.181203 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0807 11:17:12.181510 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9031,"thread_start_us":8835,"threads_started":1}
I20260807 11:17:12.184923 10864 ts_tablet_manager.cc:1918] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:12.185209 10864 log.cc:1196] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/51849447f8b843e293e898c86afc8866
I20260807 11:17:12.186082 10864 ts_tablet_manager.cc:1939] T 51849447f8b843e293e898c86afc8866 P 2c2dae56828b4222b2f13b759e771dfd: Deleting consensus metadata
I20260807 11:17:12.206601 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:12.141948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.2.223.188:33277)
0807 11:17:12.162760 (+ 20812us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:12.162766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:12.183049 (+ 20283us) server_negotiation.cc:407] Connection header received
0807 11:17:12.183276 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:12.183283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:12.183407 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:12.183621 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:12.186674 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.188243 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.190849 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.191670 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.191772 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:12.196459 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:12.196465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:12.196469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:12.196505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:12.202095 (+ 5590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.203285 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.203292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.203297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.203649 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.204265 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.204270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.204273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.204682 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:12.204968 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:12.205599 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 11:17:12.205866 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20225,"thread_start_us":20588,"threads_started":1}
I20260807 11:17:12.321693 9455 catalog_manager.cc:4841] TS defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053): tablet 51849447f8b843e293e898c86afc8866 (table test-table [id=6bfbe0d901fa4927a717d814ecdd33b0]) successfully deleted
I20260807 11:17:12.321858 9456 catalog_manager.cc:4841] TS 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595): tablet 51849447f8b843e293e898c86afc8866 (table test-table [id=6bfbe0d901fa4927a717d814ecdd33b0]) successfully deleted
I20260807 11:17:12.323056 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9436
I20260807 11:17:12.388736 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:12.367672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.2.223.188:33277)
0807 11:17:12.367909 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:12.367915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:12.371018 (+ 3103us) server_negotiation.cc:407] Connection header received
0807 11:17:12.371380 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:12.371386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:12.371470 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:12.371549 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:12.374108 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.375324 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.377423 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.378619 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.378788 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:12.381843 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:12.381851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:12.381857 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:12.381899 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:12.384891 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.385810 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.385816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.385819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.385973 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.386460 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.386467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.386494 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.387053 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:12.387296 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:12.387936 (+ 640us) server_negotiation.cc:299] Negotiation successful
0807 11:17:12.388235 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:17:12.392745 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:12.370172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.2.223.189:37579)
0807 11:17:12.370346 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:12.370351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:12.371930 (+ 1579us) server_negotiation.cc:407] Connection header received
0807 11:17:12.372237 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:12.372246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:12.372351 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:12.372437 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:12.376860 (+ 4423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.377759 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.379955 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:12.380660 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:12.380758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:12.384887 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:12.384893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:12.384898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:12.384938 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:12.389458 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.390581 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.390586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.390590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.390729 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:12.391169 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:12.391175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:12.391183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:12.391460 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:12.391732 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:12.392131 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 11:17:12.392379 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:17:13.100680 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.099940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:37063)
0807 11:17:13.100393 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.100507 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
W20260807 11:17:13.101156 10652 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:40979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.103186 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.102250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:50477)
0807 11:17:13.102835 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.103002 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":319,"threads_started":1}
W20260807 11:17:13.103652 10800 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:40979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.106810 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.105916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:34925)
0807 11:17:13.106456 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.106633 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
W20260807 11:17:13.107348 10496 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:40979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.624863 10879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7eb9c2844610439d943d7fd409bc01af)
I20260807 11:17:13.625118 10880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7eb9c2844610439d943d7fd409bc01af)
I20260807 11:17:13.625239 10879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } }
I20260807 11:17:13.625384 10880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } }
I20260807 11:17:13.626880 10880 leader_election.cc:288] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979), 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189:37579)
I20260807 11:17:13.628376 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.627067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44570)
0807 11:17:13.627812 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.628049 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":383,"threads_started":1}
I20260807 11:17:13.629366 10879 leader_election.cc:288] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979), 8b65b866775e4cdd81946944a14db9e8 (127.2.223.188:33277)
I20260807 11:17:13.629927 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.627047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44576)
0807 11:17:13.629383 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.629615 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2194,"threads_started":1}
W20260807 11:17:13.630164 10293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
W20260807 11:17:13.630877 10090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.638559 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.637975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44592)
0807 11:17:13.638162 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.638326 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:17:13.638852 10293 leader_election.cc:334] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979): Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.640259 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.639624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44604)
0807 11:17:13.639808 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.640015 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260807 11:17:13.640473 10090 leader_election.cc:334] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979): Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.641042 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.627864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:37579 (local address 127.0.0.1:43942)
0807 11:17:13.628668 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.628689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:13.628835 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:13.631281 (+ 2446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:13.631288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:13.631342 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:13.631681 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.631697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.634012 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.634021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.637077 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.637099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.640204 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.640213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.640419 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.640455 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:17:13.640552 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 11:17:13.640732 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":3,"thread_start_us":1969,"threads_started":1}
I20260807 11:17:13.641053 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.630342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.2.223.189:37579)
0807 11:17:13.630913 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:13.630918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:13.630936 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:17:13.631004 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:13.631012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:13.631087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:13.631857 (+ 770us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:13.632127 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.633795 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.637346 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.640018 (+ 2672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.640161 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.640277 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:13.640606 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 11:17:13.640786 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
I20260807 11:17:13.641782 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b65b866775e4cdd81946944a14db9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9a820e97b0704c258ffad09d42418ca2" is_pre_election: true
I20260807 11:17:13.642216 10115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b65b866775e4cdd81946944a14db9e8 in term 1.
I20260807 11:17:13.642594 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.627412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.2.223.188:33277)
0807 11:17:13.630746 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:13.630754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:13.630779 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:17:13.630860 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:13.630867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:13.630961 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:13.634530 (+ 3569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:13.635158 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.637053 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.640256 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.641517 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.641659 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.641762 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:13.642027 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 11:17:13.642203 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3181,"thread_start_us":3081,"threads_started":1}
I20260807 11:17:13.642652 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.627071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:33277 (local address 127.0.0.1:33678)
0807 11:17:13.630262 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.630280 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:13.630400 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:13.631159 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:13.631167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:13.631205 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:13.631635 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.631650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.637926 (+ 6276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.637931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.640029 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.640041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.641720 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.641725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.641860 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.641885 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:17:13.641973 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 11:17:13.642194 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"mutex_wait_us":107,"thread_start_us":2015,"threads_started":1}
I20260807 11:17:13.642975 10293 leader_election.cc:302] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b65b866775e4cdd81946944a14db9e8, 9a820e97b0704c258ffad09d42418ca2; no voters: 7eb9c2844610439d943d7fd409bc01af
I20260807 11:17:13.643410 10880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:17:13.643550 10880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7eb9c2844610439d943d7fd409bc01af)
I20260807 11:17:13.643498 10318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a820e97b0704c258ffad09d42418ca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b65b866775e4cdd81946944a14db9e8" is_pre_election: true
I20260807 11:17:13.643671 10880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:17:13.645905 10880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } }
I20260807 11:17:13.646402 10318 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a820e97b0704c258ffad09d42418ca2 in current term 2: Already voted for candidate 8b65b866775e4cdd81946944a14db9e8 in this term.
I20260807 11:17:13.646682 10880 leader_election.cc:288] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 election: Requested vote from peers 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979), 9a820e97b0704c258ffad09d42418ca2 (127.2.223.189:37579)
I20260807 11:17:13.647253 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.646757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44616)
0807 11:17:13.646934 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.647072 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 11:17:13.647372 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b65b866775e4cdd81946944a14db9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9a820e97b0704c258ffad09d42418ca2"
I20260807 11:17:13.647581 10115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:17:13.648144 10089 leader_election.cc:302] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a820e97b0704c258ffad09d42418ca2; no voters: 7eb9c2844610439d943d7fd409bc01af, 8b65b866775e4cdd81946944a14db9e8
I20260807 11:17:13.648298 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.647905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44620)
0807 11:17:13.648082 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.648166 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 11:17:13.648468 10293 leader_election.cc:334] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979): Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
I20260807 11:17:13.650054 10115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b65b866775e4cdd81946944a14db9e8 in term 2.
I20260807 11:17:13.650265 10879 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:17:13.650528 10293 leader_election.cc:302] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b65b866775e4cdd81946944a14db9e8, 9a820e97b0704c258ffad09d42418ca2; no voters: 7eb9c2844610439d943d7fd409bc01af
I20260807 11:17:13.650836 10880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:17:13.651309 10880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [term 2 LEADER]: Becoming Leader. State: Replica: 8b65b866775e4cdd81946944a14db9e8, State: Running, Role: LEADER
I20260807 11:17:13.651700 10880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } }
I20260807 11:17:13.653635 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b65b866775e4cdd81946944a14db9e8. Latest consensus state: current_term: 2 leader_uuid: "8b65b866775e4cdd81946944a14db9e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:13.654083 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:13.654814 10892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:13.658783 10892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:13.659642 10892 catalog_manager.cc:1259] Loaded cluster ID: 1c6f8d012925476e84ca5d854f65e06d
I20260807 11:17:13.659716 10892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:13.660895 10892 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:13.661723 10892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Loaded TSK: 0
I20260807 11:17:13.662748 10892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:17:13.694787 10307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:13.702018 10307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:13.707010 10115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [term 2 FOLLOWER]: Refusing update from remote peer 8b65b866775e4cdd81946944a14db9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 11:17:13.707134 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.706412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44644)
0807 11:17:13.706796 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.706905 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260807 11:17:13.707921 10890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 11:17:13.708389 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.707965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44656)
0807 11:17:13.708151 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.708241 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260807 11:17:13.708619 10293 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 -> Peer 7eb9c2844610439d943d7fd409bc01af (127.2.223.190:40979): Couldn't send request to peer 7eb9c2844610439d943d7fd409bc01af. Status: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:17:13.711107 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b65b866775e4cdd81946944a14db9e8. Latest consensus state: current_term: 2 leader_uuid: "8b65b866775e4cdd81946944a14db9e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:13.711449 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:13.713176 10880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8b65b866775e4cdd81946944a14db9e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:13.713641 10880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:13.713987 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8b65b866775e4cdd81946944a14db9e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7eb9c2844610439d943d7fd409bc01af" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40979 } } peers { permanent_uuid: "9a820e97b0704c258ffad09d42418ca2" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 37579 } attrs { promote: false } } peers { permanent_uuid: "8b65b866775e4cdd81946944a14db9e8" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 33277 } attrs { promote: false } } }
I20260807 11:17:13.714264 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a820e97b0704c258ffad09d42418ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:17:13.745622 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.723444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.2.223.131:37973)
0807 11:17:13.738902 (+ 15458us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:13.738910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:13.738935 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:17:13.739020 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:13.739024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:13.739128 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:13.739943 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:13.740176 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.741285 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.743796 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.744594 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.744758 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.744868 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:13.745176 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:17:13.745317 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15299,"thread_start_us":15179,"threads_started":1}
I20260807 11:17:13.745642 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.723346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:37973 (local address 127.0.0.1:46526)
0807 11:17:13.723539 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.723561 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:13.723667 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:13.739502 (+ 15835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:13.739507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:13.739534 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:13.739777 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.739792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.741452 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.741460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.743568 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.743584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.744787 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.744794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.744995 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.745021 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 11:17:13.745143 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 11:17:13.745282 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":29}
I20260807 11:17:13.746826 10731 tablet_service.cc:1467] Processing CreateTablet for tablet cbab0b1678e34e60b98f71d0d68926ad (DEFAULT_TABLE table=test-table [id=d5fb5c112c774beda9f105e043af3dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:13.747383 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbab0b1678e34e60b98f71d0d68926ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:13.748167 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.723693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:37595 (local address 127.0.0.1:49590)
0807 11:17:13.724042 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.724059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:13.724164 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:13.741613 (+ 17449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:13.741623 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:13.741651 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:13.742128 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.742144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.743686 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.743690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.745859 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.745874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.747334 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.747340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.747529 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.747570 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 11:17:13.747684 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:17:13.747832 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":55}
I20260807 11:17:13.748186 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.724038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.2.223.129:37595)
0807 11:17:13.741234 (+ 17196us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:13.741238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:13.741258 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:17:13.741322 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:13.741326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:13.741434 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:13.741578 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:13.742317 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.743564 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.746038 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.747044 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.747270 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.747403 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:13.747713 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 11:17:13.747857 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17081,"thread_start_us":16927,"threads_started":1}
I20260807 11:17:13.749153 10426 tablet_service.cc:1467] Processing CreateTablet for tablet cbab0b1678e34e60b98f71d0d68926ad (DEFAULT_TABLE table=test-table [id=d5fb5c112c774beda9f105e043af3dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:13.749606 10426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbab0b1678e34e60b98f71d0d68926ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:13.750065 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.724182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:33053 (local address 127.0.0.1:59958)
0807 11:17:13.740311 (+ 16129us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.740332 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:13.740443 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:13.741419 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:13.741424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:13.741458 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:13.742094 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.742112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.743609 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.743619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.747123 (+ 3504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:13.747137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.749164 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.749170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:13.749360 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.749398 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:17:13.749493 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 11:17:13.749644 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15988,"mutex_wait_us":50,"spinlock_wait_cycles":896,"thread_start_us":15055,"threads_started":1}
I20260807 11:17:13.750802 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.724581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.2.223.130:33053)
0807 11:17:13.740916 (+ 16335us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:13.740922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:13.740942 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:17:13.741011 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:13.741016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:13.741082 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:13.741183 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:13.742595 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.743459 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.747346 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:13.748780 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:13.748917 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:13.749015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:13.750289 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0807 11:17:13.750446 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16199,"thread_start_us":16126,"threads_started":1}
I20260807 11:17:13.754129 10903 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Bootstrap starting.
I20260807 11:17:13.754973 10904 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Bootstrap starting.
I20260807 11:17:13.755275 10583 tablet_service.cc:1467] Processing CreateTablet for tablet cbab0b1678e34e60b98f71d0d68926ad (DEFAULT_TABLE table=test-table [id=d5fb5c112c774beda9f105e043af3dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:13.755756 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbab0b1678e34e60b98f71d0d68926ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:13.756165 10903 tablet_bootstrap.cc:654] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:13.756879 10904 tablet_bootstrap.cc:654] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:13.759399 10904 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: No bootstrap required, opened a new log
I20260807 11:17:13.759562 10904 ts_tablet_manager.cc:1397] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:17:13.760387 10904 raft_consensus.cc:348] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.760643 10904 raft_consensus.cc:374] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:13.760740 10904 raft_consensus.cc:729] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2dae56828b4222b2f13b759e771dfd, State: Initialized, Role: FOLLOWER
I20260807 11:17:13.761051 10904 consensus_queue.cc:260] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.762319 10904 ts_tablet_manager.cc:1428] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:13.764521 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.763491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:53319)
0807 11:17:13.764062 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.764243 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"threads_started":1}
I20260807 11:17:13.766474 10495 heartbeater.cc:503] Master 127.2.223.188:33277 was elected leader, sending a full tablet report...
I20260807 11:17:13.767860 10909 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Bootstrap starting.
I20260807 11:17:13.770140 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.766942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:47687)
0807 11:17:13.769929 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.770015 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2875}
I20260807 11:17:13.770763 10909 tablet_bootstrap.cc:654] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:13.772092 10903 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: No bootstrap required, opened a new log
I20260807 11:17:13.772279 10903 ts_tablet_manager.cc:1397] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s
I20260807 11:17:13.773221 10909 tablet_bootstrap.cc:492] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: No bootstrap required, opened a new log
I20260807 11:17:13.773221 10903 raft_consensus.cc:348] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.773394 10909 ts_tablet_manager.cc:1397] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260807 11:17:13.773484 10903 raft_consensus.cc:374] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:13.773612 10903 raft_consensus.cc:729] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ff7334ca549d9a4a788d2071cb08f, State: Initialized, Role: FOLLOWER
I20260807 11:17:13.773922 10903 consensus_queue.cc:260] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.774262 10909 raft_consensus.cc:348] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.774559 10909 raft_consensus.cc:374] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:13.774668 10909 raft_consensus.cc:729] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc27725dbf4ab08e4f5646b5bc1de6, State: Initialized, Role: FOLLOWER
I20260807 11:17:13.775197 10909 consensus_queue.cc:260] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.775429 10903 ts_tablet_manager.cc:1428] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260807 11:17:13.776711 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.775218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:41363)
0807 11:17:13.776127 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.776366 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":568,"threads_started":1}
I20260807 11:17:13.777323 10799 heartbeater.cc:503] Master 127.2.223.188:33277 was elected leader, sending a full tablet report...
I20260807 11:17:13.777439 10909 ts_tablet_manager.cc:1428] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 11:17:13.778501 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.777352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:42977)
0807 11:17:13.778151 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.778268 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"threads_started":1}
I20260807 11:17:13.779644 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.778005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:43767)
0807 11:17:13.779191 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.779361 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":915,"threads_started":1}
I20260807 11:17:13.780483 10651 heartbeater.cc:503] Master 127.2.223.188:33277 was elected leader, sending a full tablet report...
I20260807 11:17:13.781553 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.781006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:36729)
0807 11:17:13.781344 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.781435 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260807 11:17:13.954113 10915 raft_consensus.cc:482] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:13.954401 10915 raft_consensus.cc:504] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.955191 10915 leader_election.cc:288] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973), 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:13.956018 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbab0b1678e34e60b98f71d0d68926ad" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ff7334ca549d9a4a788d2071cb08f" is_pre_election: true
I20260807 11:17:13.956096 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbab0b1678e34e60b98f71d0d68926ad" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2dae56828b4222b2f13b759e771dfd" is_pre_election: true
I20260807 11:17:13.956290 10753 raft_consensus.cc:2454] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 0.
I20260807 11:17:13.956434 10448 raft_consensus.cc:2454] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 0.
I20260807 11:17:13.956884 10536 leader_election.cc:302] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ff7334ca549d9a4a788d2071cb08f, defc27725dbf4ab08e4f5646b5bc1de6; no voters:
I20260807 11:17:13.957165 10915 raft_consensus.cc:2781] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:13.957296 10915 raft_consensus.cc:482] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:13.957410 10915 raft_consensus.cc:3037] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:13.959554 10915 raft_consensus.cc:504] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.960177 10915 leader_election.cc:288] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 election: Requested vote from peers 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973), 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595)
I20260807 11:17:13.960706 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbab0b1678e34e60b98f71d0d68926ad" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ff7334ca549d9a4a788d2071cb08f"
I20260807 11:17:13.960933 10753 raft_consensus.cc:3037] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:13.960872 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbab0b1678e34e60b98f71d0d68926ad" candidate_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2dae56828b4222b2f13b759e771dfd"
I20260807 11:17:13.961117 10448 raft_consensus.cc:3037] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:13.963363 10753 raft_consensus.cc:2454] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 1.
I20260807 11:17:13.963442 10448 raft_consensus.cc:2454] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc27725dbf4ab08e4f5646b5bc1de6 in term 1.
I20260807 11:17:13.963835 10536 leader_election.cc:302] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ff7334ca549d9a4a788d2071cb08f, defc27725dbf4ab08e4f5646b5bc1de6; no voters:
I20260807 11:17:13.964192 10915 raft_consensus.cc:2781] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:13.964696 10915 raft_consensus.cc:686] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 LEADER]: Becoming Leader. State: Replica: defc27725dbf4ab08e4f5646b5bc1de6, State: Running, Role: LEADER
I20260807 11:17:13.965037 10915 consensus_queue.cc:237] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } }
I20260807 11:17:13.965965 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:13.965454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:58895)
0807 11:17:13.965657 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:13.965786 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 11:17:13.967976 10308 catalog_manager.cc:5495] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc27725dbf4ab08e4f5646b5bc1de6" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 33053 } health_report { overall_health: HEALTHY } } }
I20260807 11:17:14.030301 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.025330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.2.223.130:33053)
0807 11:17:14.025544 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:14.025552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:14.025592 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:17:14.025809 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:14.025814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:14.025878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:14.025982 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:14.026409 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.027217 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.028592 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.029027 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.029169 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:14.029560 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:14.029688 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:14.029791 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:14.029956 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 11:17:14.030039 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:17:14.033663 10448 raft_consensus.cc:1261] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Refusing update from remote peer defc27725dbf4ab08e4f5646b5bc1de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:14.033787 10753 raft_consensus.cc:1261] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Refusing update from remote peer defc27725dbf4ab08e4f5646b5bc1de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:14.034410 10918 consensus_queue.cc:1035] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2dae56828b4222b2f13b759e771dfd" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 37595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:14.034796 10915 consensus_queue.cc:1035] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495ff7334ca549d9a4a788d2071cb08f" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:14.037779 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.036850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:52193)
0807 11:17:14.037188 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.037442 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260807 11:17:14.038430 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.037831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:43097)
0807 11:17:14.038084 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.038190 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260807 11:17:14.044950 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.043989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:57327)
0807 11:17:14.044664 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.044764 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560}
I20260807 11:17:14.045450 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.044954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.129:40033)
0807 11:17:14.045200 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.045287 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260807 11:17:14.046415 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.045880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.131:37365)
0807 11:17:14.046127 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.046221 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260807 11:17:14.049266 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.048720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.2.223.130:45869)
0807 11:17:14.049006 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.049096 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260807 11:17:14.051496 10307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:14.051819 10307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:14.054641 10922 mvcc.cc:205] Tried to move back new op lower bound from 7315871473795342336 to 7315871473523523584. Current Snapshot: MvccSnapshot[applied={T|T < 7315871473795342336 or (T in {7315871473795342336})}]
I20260807 11:17:14.059325 10307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b65b866775e4cdd81946944a14db9e8: Sending DeleteTablet for 3 replicas of tablet cbab0b1678e34e60b98f71d0d68926ad
I20260807 11:17:14.060442 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet cbab0b1678e34e60b98f71d0d68926ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:14 UTC) from {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:46526
I20260807 11:17:14.060664 10583 tablet_service.cc:1514] Processing DeleteTablet for tablet cbab0b1678e34e60b98f71d0d68926ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:14 UTC) from {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:59958
I20260807 11:17:14.060664 10426 tablet_service.cc:1514] Processing DeleteTablet for tablet cbab0b1678e34e60b98f71d0d68926ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:14 UTC) from {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:49590
I20260807 11:17:14.060762 10425 tablet_service.cc:1514] Processing DeleteTablet for tablet cbab0b1678e34e60b98f71d0d68926ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:14 UTC) from {username='oryx', principal='oryx/127.2.223.188@KRBTEST.COM'} at 127.0.0.1:49590
I20260807 11:17:14.064648 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 10278
I20260807 11:17:14.071985 10929 tablet_replica.cc:330] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: stopping tablet replica
I20260807 11:17:14.072269 10929 raft_consensus.cc:2229] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:14.072683 10929 raft_consensus.cc:2258] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:14.073424 10930 tablet_replica.cc:330] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: stopping tablet replica
I20260807 11:17:14.073700 10930 raft_consensus.cc:2229] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:14.074239 10930 raft_consensus.cc:2258] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:14.074982 10931 tablet_replica.cc:330] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: stopping tablet replica
I20260807 11:17:14.075201 10931 raft_consensus.cc:2229] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:14.075518 10931 raft_consensus.cc:2258] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:14.075857 10930 ts_tablet_manager.cc:1905] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:14.076587 10929 ts_tablet_manager.cc:1905] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:14.078202 10931 ts_tablet_manager.cc:1905] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:14.082556 10930 ts_tablet_manager.cc:1918] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:14.082698 10930 log.cc:1196] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/cbab0b1678e34e60b98f71d0d68926ad
I20260807 11:17:14.082775 10931 ts_tablet_manager.cc:1918] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:14.082912 10931 log.cc:1196] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/cbab0b1678e34e60b98f71d0d68926ad
I20260807 11:17:14.083070 10929 ts_tablet_manager.cc:1918] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:14.083184 10930 ts_tablet_manager.cc:1939] T cbab0b1678e34e60b98f71d0d68926ad P defc27725dbf4ab08e4f5646b5bc1de6: Deleting consensus metadata
I20260807 11:17:14.083213 10929 log.cc:1196] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/cbab0b1678e34e60b98f71d0d68926ad
I20260807 11:17:14.083405 10931 ts_tablet_manager.cc:1939] T cbab0b1678e34e60b98f71d0d68926ad P 2c2dae56828b4222b2f13b759e771dfd: Deleting consensus metadata
I20260807 11:17:14.083678 10929 ts_tablet_manager.cc:1939] T cbab0b1678e34e60b98f71d0d68926ad P 495ff7334ca549d9a4a788d2071cb08f: Deleting consensus metadata
I20260807 11:17:14.084460 10293 catalog_manager.cc:4841] TS defc27725dbf4ab08e4f5646b5bc1de6 (127.2.223.130:33053): tablet cbab0b1678e34e60b98f71d0d68926ad (table test-table [id=d5fb5c112c774beda9f105e043af3dfa]) successfully deleted
I20260807 11:17:14.084686 10294 catalog_manager.cc:4841] TS 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595): tablet cbab0b1678e34e60b98f71d0d68926ad (table test-table [id=d5fb5c112c774beda9f105e043af3dfa]) successfully deleted
I20260807 11:17:14.084962 10294 catalog_manager.cc:4841] TS 495ff7334ca549d9a4a788d2071cb08f (127.2.223.131:37973): tablet cbab0b1678e34e60b98f71d0d68926ad (table test-table [id=d5fb5c112c774beda9f105e043af3dfa]) successfully deleted
W20260807 11:17:14.085855 10294 catalog_manager.cc:4808] TS 2c2dae56828b4222b2f13b759e771dfd (127.2.223.129:37595): delete failed for tablet cbab0b1678e34e60b98f71d0d68926ad because the tablet was not found. No further retry: Not found: Tablet not found: cbab0b1678e34e60b98f71d0d68926ad
I20260807 11:17:14.093755 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.066519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.2.223.129:37595)
0807 11:17:14.066825 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:14.066834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:14.075542 (+ 8708us) server_negotiation.cc:407] Connection header received
0807 11:17:14.075618 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:14.075623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:14.075720 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:14.075862 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:14.077308 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.078356 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.084911 (+ 6555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.085439 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.085593 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:14.088497 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:14.088510 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:14.088525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:14.088557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:14.091404 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.092018 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.092025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.092027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.092165 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.092488 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.092491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.092493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.092836 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:14.093059 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:14.093352 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 11:17:14.093517 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:17:14.094525 10490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:14.123657 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.123159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44660)
0807 11:17:14.123385 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.123487 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 11:17:14.124756 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.124369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40979 (local address 127.0.0.1:44664)
0807 11:17:14.124546 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:14.124628 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 11:17:14.191203 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 10278
I20260807 11:17:14.215314 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 10507
I20260807 11:17:14.229415 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.216091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.2.223.130:33053)
0807 11:17:14.216285 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:14.216294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:14.216379 (+ 85us) server_negotiation.cc:407] Connection header received
0807 11:17:14.216604 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:14.216615 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:14.216680 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:14.216788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:14.218160 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.219029 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.220278 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.220695 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.220834 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:14.223502 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:14.223518 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:14.223535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:14.223597 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:14.226359 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.227242 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.227254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.227262 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.227416 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.227807 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.227813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.227816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.228272 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:14.228591 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:14.228865 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 11:17:14.229086 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:17:14.230158 10646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:14.354801 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 10507
I20260807 11:17:14.382881 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 10664
I20260807 11:17:14.396265 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:14.383693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.2.223.131:37973)
0807 11:17:14.384216 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:14.384220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:14.384237 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:17:14.384298 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:14.384302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:14.384370 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:14.384478 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:14.385813 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.386579 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.387826 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:14.388290 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:14.388418 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:14.390993 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:14.391021 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:14.391039 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:14.391074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:14.393789 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.394425 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.394433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.394435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.394648 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:14.394983 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:14.394986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:14.394988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:14.395344 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:14.395587 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:14.395799 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 11:17:14.395972 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260807 11:17:14.397080 10794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:14.492236 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 10664
I20260807 11:17:14.515379 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 9958
I20260807 11:17:14.516708 10139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:14.640908 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 9958
I20260807 11:17:14.664641 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 10068
I20260807 11:17:14.665795 10339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:14.793404 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 10068
2026-08-07T11:17:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10802 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:14 dist-test-slave-5l3k krb5kdc[10943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:14 dist-test-slave-5l3k krb5kdc[10943](info): set up 2 sockets
Aug 07 11:17:14 dist-test-slave-5l3k krb5kdc[10943](info): commencing operation
krb5kdc: starting...
W20260807 11:17:15.956741 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:16 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab.
Entry for principal oryx/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab.
I20260807 11:17:16.234166 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:37227
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab
--principal=oryx/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:37227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:16.581339 10956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:16.581804 10956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:16.581974 10956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:16.589188 10956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:16.589340 10956 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:17:16.589437 10956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:16.589536 10956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:16.589636 10956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:16.589728 10956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:16.599884 10956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:37227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.190.keytab
--principal=oryx/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:16.601238 10956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:16.602727 10956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:16.614681 10961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:16.614730 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:16.614710 10962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:16.615641 10956 server_base.cc:1034] running on GCE node
I20260807 11:17:16.616447 10956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:16.617820 10956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:16.619067 10956 hybrid_clock.cc:648] HybridClock initialized: now 1786101436618984 us; error 61 us; skew 500 ppm
I20260807 11:17:16.619397 10956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:16 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101436, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:16.622663 10956 init.cc:377] Logged in from keytab as oryx/127.2.223.190@KRBTEST.COM (short username oryx)
I20260807 11:17:16.624097 10956 webserver.cc:460] Webserver started at http://127.2.223.190:35669/ using document root <none> and password file <none>
I20260807 11:17:16.624749 10956 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:16.624840 10956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:16.625121 10956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:16.628073 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "f27650aa94dc40ad9a4c671c8688da40"
format_stamp: "Formatted at 2026-08-07 11:17:16 on dist-test-slave-5l3k"
server_key: "39974001dbcc3754c80fc7d0c8933915"
server_key_iv: "10d9a75acb737ae6dd6a7be993ee168e"
server_key_version: "encryptionkey@0"
I20260807 11:17:16.628845 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "f27650aa94dc40ad9a4c671c8688da40"
format_stamp: "Formatted at 2026-08-07 11:17:16 on dist-test-slave-5l3k"
server_key: "39974001dbcc3754c80fc7d0c8933915"
server_key_iv: "10d9a75acb737ae6dd6a7be993ee168e"
server_key_version: "encryptionkey@0"
I20260807 11:17:16.633908 10956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:17:16.637455 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:16.638909 10956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:16.639112 10956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "f27650aa94dc40ad9a4c671c8688da40"
format_stamp: "Formatted at 2026-08-07 11:17:16 on dist-test-slave-5l3k"
server_key: "39974001dbcc3754c80fc7d0c8933915"
server_key_iv: "10d9a75acb737ae6dd6a7be993ee168e"
server_key_version: "encryptionkey@0"
I20260807 11:17:16.639330 10956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:16.663069 10956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:16.667301 10956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:16.667603 10956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:16.691057 10956 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:37227
I20260807 11:17:16.691038 11035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:37227 every 8 connection(s)
I20260807 11:17:16.692471 10956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:16.693846 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 10956
I20260807 11:17:16.694056 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:16.694805 2942 external_mini_cluster.cc:1428] Setting key 13bd6a2bf1e61d7ee225edfae2b9133f
I20260807 11:17:16.696920 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:17:16 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:16.708599 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Bootstrap starting.
I20260807 11:17:16.712054 11036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:16.713142 11036 log.cc:824] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:16.716572 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: No bootstrap required, opened a new log
I20260807 11:17:16.717681 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:16.696995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.2.223.190:37227)
0807 11:17:16.697781 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:16.697792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:16.697832 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:17:16.699544 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:16.699567 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:16.699953 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:16.700373 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:16.701851 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:16.703274 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:16.704538 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:16.705226 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:16.705432 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:16.709689 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:16.709697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:16.709711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:16.709754 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:16.712952 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:16.713772 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:16.713784 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:16.713792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:16.713991 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:16.714484 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:16.714489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:16.714492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:16.715198 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:16.715508 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:16.715873 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 11:17:16.716311 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":362,"threads_started":1}
I20260807 11:17:16.725390 11036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } }
I20260807 11:17:16.725710 11036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:16.725785 11036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27650aa94dc40ad9a4c671c8688da40, State: Initialized, Role: FOLLOWER
I20260807 11:17:16.726321 11036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } }
I20260807 11:17:16.726586 11036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:16.726718 11036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:16.726877 11036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:16.729254 11036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } }
I20260807 11:17:16.729703 11036 leader_election.cc:302] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27650aa94dc40ad9a4c671c8688da40; no voters:
I20260807 11:17:16.730271 11036 leader_election.cc:288] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:16.730414 11041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:16.731256 11041 raft_consensus.cc:686] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [term 1 LEADER]: Becoming Leader. State: Replica: f27650aa94dc40ad9a4c671c8688da40, State: Running, Role: LEADER
I20260807 11:17:16.731873 11041 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } }
I20260807 11:17:16.732436 11036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:16.734892 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27650aa94dc40ad9a4c671c8688da40. Latest consensus state: current_term: 1 leader_uuid: "f27650aa94dc40ad9a4c671c8688da40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } } }
I20260807 11:17:16.734939 11042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f27650aa94dc40ad9a4c671c8688da40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27650aa94dc40ad9a4c671c8688da40" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 37227 } } }
I20260807 11:17:16.735277 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:16.735333 11042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:16.739151 11050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:16.748366 11050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:16.758195 11050 catalog_manager.cc:1347] Generated new cluster ID: 0a2cde8b2f554607813f8cfe31c0e57d
I20260807 11:17:16.758333 11050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:16.777309 11050 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:16.778731 11050 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:16.789798 11050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Generated new TSK 0
I20260807 11:17:16.790663 11050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
I20260807 11:17:16.854594 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:37227
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:17.199044 11063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:17.199337 11063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:17.199437 11063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:17.207293 11063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:17.207407 11063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:17.207499 11063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:17:17.207580 11063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:17.218011 11063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:37227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:17.219437 11063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:17.220852 11063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:17.234935 11070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:17.235131 11068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:17.235636 11072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:17.236572 11063 server_base.cc:1034] running on GCE node
I20260807 11:17:17.237257 11063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:17.238291 11063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:17.239547 11063 hybrid_clock.cc:648] HybridClock initialized: now 1786101437239456 us; error 63 us; skew 500 ppm
I20260807 11:17:17.239878 11063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:17 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:17.243422 11063 init.cc:377] Logged in from keytab as oryx/127.2.223.129@KRBTEST.COM (short username oryx)
I20260807 11:17:17.244910 11063 webserver.cc:460] Webserver started at http://127.2.223.129:34561/ using document root <none> and password file <none>
I20260807 11:17:17.245648 11063 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:17.245785 11063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:17.246132 11063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:17.249199 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "4d10d0a809af40e4bc146614d8b208a8"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "baaa0e6c2e08bd82b444ef5eb2aafab5"
server_key_iv: "be17393207b1c99e19c401733adb250a"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.250026 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "4d10d0a809af40e4bc146614d8b208a8"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "baaa0e6c2e08bd82b444ef5eb2aafab5"
server_key_iv: "be17393207b1c99e19c401733adb250a"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.255412 11063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:17:17.258975 11079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:17.260241 11063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:17:17.260416 11063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "4d10d0a809af40e4bc146614d8b208a8"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "baaa0e6c2e08bd82b444ef5eb2aafab5"
server_key_iv: "be17393207b1c99e19c401733adb250a"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.260641 11063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:17.283006 11063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:17.287196 11063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:17.287485 11063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:17.290349 11063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:17.290499 11063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:17.290642 11063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:17.290719 11063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:17:17 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:17.316726 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.291986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.129:60921)
0807 11:17:17.292809 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:17.292846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:17.294819 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:17.295647 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:17.295663 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:17.296511 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:17.297435 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.297467 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.299286 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.299292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.301583 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.301597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.302699 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.302710 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.303108 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.304117 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:17.304164 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:17.307732 (+ 3568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:17.310877 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.310886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.310903 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.311519 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.311969 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.311976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.311984 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.312284 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.313182 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:17.313188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:17.313800 (+ 612us) client_negotiation.cc:769] Sending connection context
0807 11:17:17.314215 (+ 415us) client_negotiation.cc:240] Negotiation successful
0807 11:17:17.314884 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":350,"threads_started":1}
I20260807 11:17:17.317399 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.291144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:60921 (local address 127.2.223.190:37227)
0807 11:17:17.291685 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:17.291700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:17.292963 (+ 1263us) server_negotiation.cc:407] Connection header received
0807 11:17:17.295118 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:17.295125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:17.295218 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:17.295352 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:17.297694 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.299062 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.301843 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.302374 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.302554 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.307985 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:17.307990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:17.307993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:17.308029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:17.310727 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.311715 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.311720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.311731 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.311848 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.312502 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.312507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.312509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.312950 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:17.313113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:17.316828 (+ 3715us) server_negotiation.cc:299] Negotiation successful
0807 11:17:17.317112 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
I20260807 11:17:17.340134 11063 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:44971
I20260807 11:17:17.340144 11204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:44971 every 8 connection(s)
I20260807 11:17:17.341665 11063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:17.343444 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11063
I20260807 11:17:17.343678 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:17.344164 11205 heartbeater.cc:347] Connected to a master server at 127.2.223.190:37227
I20260807 11:17:17.344375 11205 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:17.344477 2942 external_mini_cluster.cc:1428] Setting key 90802446042297a89e6ec5749880d09f
I20260807 11:17:17.345099 11205 heartbeater.cc:511] Master 127.2.223.190:37227 requested a full tablet report, sending...
I20260807 11:17:17.347383 10988 ts_manager.cc:194] Registered new tserver with Master: 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971)
I20260807 11:17:17.348917 10988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:60921
I20260807 11:17:17.358542 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.323530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.129:48995)
0807 11:17:17.338385 (+ 14855us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:17.338403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:17.338655 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:17.339347 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:17.339354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:17.340184 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:17.340661 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.340675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.341711 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.341717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.345470 (+ 3753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.345490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.350592 (+ 5102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.350605 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.350843 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.351630 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:17.351669 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:17.352819 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:17.355813 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.355821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.355827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.356417 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.356809 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.356816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.356822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.356995 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.357504 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:17.357512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:17.357744 (+ 232us) client_negotiation.cc:769] Sending connection context
0807 11:17:17.357920 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 11:17:17.358167 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14712,"thread_start_us":15250,"threads_started":1}
I20260807 11:17:17.358778 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.323516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:48995 (local address 127.2.223.190:37227)
0807 11:17:17.323768 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:17.323774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:17.338546 (+ 14772us) server_negotiation.cc:407] Connection header received
0807 11:17:17.338834 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:17.338843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:17.338949 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:17.339111 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:17.340835 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.341534 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.345638 (+ 4104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.346362 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.346529 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.353022 (+ 6493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:17.353027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:17.353030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:17.353062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:17.355671 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.356551 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.356555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.356557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.356695 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.357114 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.357117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.357119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.357386 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:17.357507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:17.358081 (+ 574us) server_negotiation.cc:299] Negotiation successful
0807 11:17:17.358349 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
I20260807 11:17:17.406062 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:37227
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:17.744722 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:17.745127 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:17.745321 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:17.753208 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:17.753365 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:17.753533 11209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:17:17.753695 11209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:17.763808 11209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:37227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:17.765276 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:17.766729 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:17.780190 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:17.780124 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:17.780164 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:17.780916 11209 server_base.cc:1034] running on GCE node
I20260807 11:17:17.781481 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:17.782411 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:17.783669 11209 hybrid_clock.cc:648] HybridClock initialized: now 1786101437783607 us; error 43 us; skew 500 ppm
I20260807 11:17:17.783927 11209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:17 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:17.786966 11209 init.cc:377] Logged in from keytab as oryx/127.2.223.130@KRBTEST.COM (short username oryx)
I20260807 11:17:17.788434 11209 webserver.cc:460] Webserver started at http://127.2.223.130:44797/ using document root <none> and password file <none>
I20260807 11:17:17.789093 11209 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:17.789183 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:17.789455 11209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:17.792546 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "7bd1ce5ccc10439b985c56442c3ae442"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "19104489f8156695bfd4c6044e278a11"
server_key_iv: "6a5a82d56ae0c567eca69f4ddf564b92"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.793294 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd1ce5ccc10439b985c56442c3ae442"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "19104489f8156695bfd4c6044e278a11"
server_key_iv: "6a5a82d56ae0c567eca69f4ddf564b92"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.798331 11209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:17:17.801817 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:17.803256 11209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:17.803387 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "7bd1ce5ccc10439b985c56442c3ae442"
format_stamp: "Formatted at 2026-08-07 11:17:17 on dist-test-slave-5l3k"
server_key: "19104489f8156695bfd4c6044e278a11"
server_key_iv: "6a5a82d56ae0c567eca69f4ddf564b92"
server_key_version: "encryptionkey@0"
I20260807 11:17:17.803578 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:17.820968 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:17.825845 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:17.826074 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:17.828986 11209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:17.829110 11209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:17.829262 11209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:17.829331 11209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:17:17 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101437, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:17.853858 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.829415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:40773 (local address 127.2.223.190:37227)
0807 11:17:17.829715 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:17.829723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:17.831173 (+ 1450us) server_negotiation.cc:407] Connection header received
0807 11:17:17.833500 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:17.833507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:17.833608 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:17.833725 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:17.836098 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.837168 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.839686 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.840218 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.840360 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.844497 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:17.844507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:17.844516 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:17.844572 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:17.848890 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.849762 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.849769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.849775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.849963 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.850505 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.850513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.850519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.850995 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:17.851192 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:17.852959 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0807 11:17:17.853304 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260807 11:17:17.854645 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.830154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.130:40773)
0807 11:17:17.831012 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:17.831051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:17.833190 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:17.834026 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:17.834034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:17.834885 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:17.835830 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.835863 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.837339 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.837348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.839533 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.839546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.840467 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.840477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.840719 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.841483 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:17.841505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:17.844212 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:17.849061 (+ 4849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.849077 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.849111 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.849612 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.850122 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.850129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.850132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.850342 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.851298 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:17.851310 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:17.851913 (+ 603us) client_negotiation.cc:769] Sending connection context
0807 11:17:17.852327 (+ 414us) client_negotiation.cc:240] Negotiation successful
0807 11:17:17.853006 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260807 11:17:17.882369 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.864009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:38413 (local address 127.2.223.190:37227)
0807 11:17:17.864276 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:17.864300 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:17.865015 (+ 715us) server_negotiation.cc:407] Connection header received
0807 11:17:17.865332 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:17.865341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:17.865437 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:17.865575 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:17.866910 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.868223 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.871018 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.871862 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.872055 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.874701 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:17.874706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:17.874710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:17.874749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:17.878285 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.879576 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.879582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.879585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.879792 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:17.880329 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:17.880334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:17.880336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:17.880691 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:17.880824 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:17.881563 (+ 739us) server_negotiation.cc:299] Negotiation successful
0807 11:17:17.881874 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:17:17.883394 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:17.863976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.130:38413)
0807 11:17:17.864859 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:17.864883 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:17.865116 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:17.865656 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:17.865664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:17.866254 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:17.866708 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.866730 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.868444 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.868449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.870783 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:17.870818 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:17.872132 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:17.872146 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:17.872427 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:17.873433 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:17.873469 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:17.874411 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:17.878515 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.878521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.878525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.879384 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.879964 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:17.879972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:17.879977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:17.880177 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:17.880830 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:17.880838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:17.881159 (+ 321us) client_negotiation.cc:769] Sending connection context
0807 11:17:17.881366 (+ 207us) client_negotiation.cc:240] Negotiation successful
0807 11:17:17.882114 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":495,"threads_started":1}
I20260807 11:17:17.889214 11209 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:35073
I20260807 11:17:17.889250 11349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:35073 every 8 connection(s)
I20260807 11:17:17.890447 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:17.892746 11350 heartbeater.cc:347] Connected to a master server at 127.2.223.190:37227
I20260807 11:17:17.892992 11350 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:17.893687 11350 heartbeater.cc:511] Master 127.2.223.190:37227 requested a full tablet report, sending...
I20260807 11:17:17.894418 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11209
I20260807 11:17:17.894634 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:17.895210 2942 external_mini_cluster.cc:1428] Setting key 333a6ea3d23f4cbf95feec2e640da03b
I20260807 11:17:17.895328 10988 ts_manager.cc:194] Registered new tserver with Master: 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073)
I20260807 11:17:17.896348 10988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:40773
WARNING: no policy specified for oryx/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
I20260807 11:17:17.958940 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:37227
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:18.299295 11355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:18.299743 11355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:18.299916 11355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:18.307652 11355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:18.307802 11355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:18.307950 11355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:17:18.308053 11355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:18.318293 11355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.223.190:37227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:18.319752 11355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:18.321220 11355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:18.335675 11362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:18.335634 11360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:18.335716 11364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:18.337185 11355 server_base.cc:1034] running on GCE node
I20260807 11:17:18.337949 11355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:18.339258 11355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:18.340556 11355 hybrid_clock.cc:648] HybridClock initialized: now 1786101438340416 us; error 95 us; skew 500 ppm
I20260807 11:17:18.340957 11355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:18 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101438, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:18.345191 11355 init.cc:377] Logged in from keytab as oryx/127.2.223.131@KRBTEST.COM (short username oryx)
I20260807 11:17:18.347152 11355 webserver.cc:460] Webserver started at http://127.2.223.131:36069/ using document root <none> and password file <none>
I20260807 11:17:18.347893 11355 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:18.348030 11355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:18.348378 11355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:18.351617 11355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "a3de5b4460a14551b9b216747223f98e"
format_stamp: "Formatted at 2026-08-07 11:17:18 on dist-test-slave-5l3k"
server_key: "4db35a9def0f26bc5957c410bc7dc11f"
server_key_iv: "0b5a0f40c2b14314a38d52940dda332f"
server_key_version: "encryptionkey@0"
I20260807 11:17:18.352264 11205 heartbeater.cc:503] Master 127.2.223.190:37227 was elected leader, sending a full tablet report...
I20260807 11:17:18.352535 11355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "a3de5b4460a14551b9b216747223f98e"
format_stamp: "Formatted at 2026-08-07 11:17:18 on dist-test-slave-5l3k"
server_key: "4db35a9def0f26bc5957c410bc7dc11f"
server_key_iv: "0b5a0f40c2b14314a38d52940dda332f"
server_key_version: "encryptionkey@0"
I20260807 11:17:18.358363 11355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:18.361984 11371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:18.363524 11355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:18.363724 11355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "a3de5b4460a14551b9b216747223f98e"
format_stamp: "Formatted at 2026-08-07 11:17:18 on dist-test-slave-5l3k"
server_key: "4db35a9def0f26bc5957c410bc7dc11f"
server_key_iv: "0b5a0f40c2b14314a38d52940dda332f"
server_key_version: "encryptionkey@0"
I20260807 11:17:18.363958 11355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:18.402796 11355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:18.407639 11355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:18.407891 11355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:18.410766 11355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:18.410856 11355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:18.411003 11355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:18.411063 11355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:17:18 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101438, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:18.437013 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:18.411557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:54605 (local address 127.2.223.190:37227)
0807 11:17:18.412200 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:18.412208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:18.413424 (+ 1216us) server_negotiation.cc:407] Connection header received
0807 11:17:18.415963 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:18.415972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:18.416086 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:18.416253 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:18.418633 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.420059 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.422646 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.423125 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.423284 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:18.428629 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:18.428639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:18.428648 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:18.428725 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:18.433182 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:18.434148 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:18.434157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:18.434163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:18.434362 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:18.435144 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:18.435148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:18.435150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:18.435435 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:18.435615 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:18.436374 (+ 759us) server_negotiation.cc:299] Negotiation successful
0807 11:17:18.436611 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
I20260807 11:17:18.438208 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:18.412258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.131:54605)
0807 11:17:18.413232 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:18.413280 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:18.415584 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:18.416593 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:18.416629 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:18.417443 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:18.418386 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:18.418423 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.420275 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.420284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:18.422447 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:18.422461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.423426 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.423438 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:18.423890 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:18.424887 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:18.424931 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:18.428329 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:18.433407 (+ 5078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:18.433423 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:18.433448 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:18.433946 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:18.434552 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:18.434559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:18.434581 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:18.434969 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:18.435575 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:18.435584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:18.435984 (+ 400us) client_negotiation.cc:769] Sending connection context
0807 11:17:18.436268 (+ 284us) client_negotiation.cc:240] Negotiation successful
0807 11:17:18.436838 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":417,"threads_started":1}
I20260807 11:17:18.462148 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:18.445108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:43245 (local address 127.2.223.190:37227)
0807 11:17:18.445317 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:18.445324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:18.446084 (+ 760us) server_negotiation.cc:407] Connection header received
0807 11:17:18.446326 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:18.446333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:18.446413 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:18.446553 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:18.448258 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.449242 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.451896 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.452308 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.452414 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:18.455429 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:18.455437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:18.455444 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:18.455493 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:18.458911 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:18.459908 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:18.459913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:18.459915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:18.460075 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:18.460586 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:18.460593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:18.460598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:18.460925 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:18.461113 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:18.461532 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 11:17:18.461770 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:17:18.462255 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:18.445139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.2.223.131:43245)
0807 11:17:18.445938 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:18.445964 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:18.446168 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:18.446610 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:18.446618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:18.447446 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:18.448046 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:18.448072 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.449392 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.449398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:18.451715 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:18.451735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.452530 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.452539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:18.452773 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:18.453570 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:18.453601 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:18.455109 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:18.459086 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:18.459095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:18.459105 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:18.459740 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:18.460210 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:18.460216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:18.460220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:18.460426 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:18.461065 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:18.461070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:18.461262 (+ 192us) client_negotiation.cc:769] Sending connection context
0807 11:17:18.461414 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 11:17:18.461791 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":395,"threads_started":1}
I20260807 11:17:18.471457 11355 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:41139
I20260807 11:17:18.471488 11497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:41139 every 8 connection(s)
I20260807 11:17:18.472823 11355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:18.475162 11498 heartbeater.cc:347] Connected to a master server at 127.2.223.190:37227
I20260807 11:17:18.475363 11498 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:18.476006 11498 heartbeater.cc:511] Master 127.2.223.190:37227 requested a full tablet report, sending...
I20260807 11:17:18.477402 10987 ts_manager.cc:194] Registered new tserver with Master: a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139)
I20260807 11:17:18.478338 10987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:54605
I20260807 11:17:18.480316 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11355
I20260807 11:17:18.480486 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:18.480952 2942 external_mini_cluster.cc:1428] Setting key 679970b7c5250c96737dee3a9657eb35
I20260807 11:17:18.484089 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:17:18 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:17:18 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:18 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:18.899224 11350 heartbeater.cc:503] Master 127.2.223.190:37227 was elected leader, sending a full tablet report...
I20260807 11:17:19.481150 11498 heartbeater.cc:503] Master 127.2.223.190:37227 was elected leader, sending a full tablet report...
W20260807 11:17:19.664458 11506 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.120s user 0.007s sys 0.014s
W20260807 11:17:19.665830 11380 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:18.538222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.2.223.190:37227)
0807 11:17:18.538415 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:18.538420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:18.538679 (+ 259us) server_negotiation.cc:407] Connection header received
0807 11:17:18.538848 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:18.538853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:18.538960 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:18.539109 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:18.540585 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.541534 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.542820 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:18.543367 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:18.543491 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.665581 (+1122090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51076: BlockingRecv error: recv got EOF from 127.0.0.1:51076 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 11:17:19 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:19.687620 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.669857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.2.223.190:37227)
0807 11:17:19.670029 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:19.670034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:19.670443 (+ 409us) server_negotiation.cc:407] Connection header received
0807 11:17:19.670667 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:19.670671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:19.670739 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:19.670827 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:19.672038 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.672750 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.673927 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.674361 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.674464 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.682761 (+ 8297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:19.682766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:19.682770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:19.682804 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:19.685490 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:19.686061 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:19.686065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:19.686067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:19.686193 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:19.686554 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:19.686557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:19.686559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:19.686808 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:19.687012 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:19.687209 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 11:17:19.687372 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:17:19.691658 10987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:19.699124 10987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:19.736726 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.713655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.2.223.131:41139)
0807 11:17:19.727108 (+ 13453us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:19.727121 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:19.727719 (+ 598us) server_negotiation.cc:407] Connection header received
0807 11:17:19.727890 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:19.727901 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:19.728189 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:19.728412 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:19.729242 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.730928 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.733756 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735250 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735387 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.735591 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:19.736229 (+ 638us) server_negotiation.cc:299] Negotiation successful
0807 11:17:19.736382 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13206,"thread_start_us":13056,"threads_started":1}
I20260807 11:17:19.736783 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.711949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.0.0.1:47238)
0807 11:17:19.719596 (+ 7647us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:19.719656 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:19.719875 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:19.728634 (+ 8759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:19.728639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:19.728701 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:19.729286 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.729298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.731128 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.731137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.733505 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.733518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735799 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.736081 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.736115 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:17:19.736221 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 11:17:19.736362 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6879,"mutex_wait_us":110,"spinlock_wait_cycles":1408,"thread_start_us":326,"threads_started":1}
I20260807 11:17:19.737000 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.712980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.2.223.129:44971)
0807 11:17:19.728280 (+ 15300us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:19.728291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:19.728318 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:19.728421 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:19.728428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:19.728642 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:19.729227 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:19.730097 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.731195 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.733908 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735076 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735228 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.735447 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:19.736479 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0807 11:17:19.736690 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15139,"thread_start_us":14958,"threads_started":1}
I20260807 11:17:19.737018 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.712179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.2.223.130:35073)
0807 11:17:19.727740 (+ 15561us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:19.727749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:19.727780 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:17:19.727896 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:19.727907 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:19.728189 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:19.728804 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:19.729484 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.730972 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.734092 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735528 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735683 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.735876 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:19.736507 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 11:17:19.736693 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15390,"thread_start_us":16162,"threads_started":1}
I20260807 11:17:19.736368 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.712742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44971 (local address 127.0.0.1:57158)
0807 11:17:19.725324 (+ 12582us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:19.725349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:19.725489 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:19.729467 (+ 3978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:19.729471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:19.729491 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:19.729904 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.729917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.731342 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.731347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.733762 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.733773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735280 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.735466 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.735519 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 11:17:19.735680 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:17:19.735888 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12406,"mutex_wait_us":2,"thread_start_us":12254,"threads_started":1}
I20260807 11:17:19.737715 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:19.713104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41139 (local address 127.0.0.1:60114)
0807 11:17:19.727580 (+ 14476us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:19.727598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:19.727720 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:19.728432 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:19.728438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:19.728467 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:19.728981 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.728995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.731077 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.731082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.733603 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:19.733619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:19.735776 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:19.735783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:19.736013 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:19.736065 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 11:17:19.736183 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 11:17:19.736303 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14363,"mutex_wait_us":95,"thread_start_us":14198,"threads_started":1}
I20260807 11:17:19.741230 11424 tablet_service.cc:1467] Processing CreateTablet for tablet a494181193424f02a2e6db700ad917b0 (DEFAULT_TABLE table=test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:19.741245 11279 tablet_service.cc:1467] Processing CreateTablet for tablet a494181193424f02a2e6db700ad917b0 (DEFAULT_TABLE table=test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:19.742025 11133 tablet_service.cc:1467] Processing CreateTablet for tablet a494181193424f02a2e6db700ad917b0 (DEFAULT_TABLE table=test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:19.742705 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a494181193424f02a2e6db700ad917b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:19.742707 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a494181193424f02a2e6db700ad917b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:19.743322 11133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a494181193424f02a2e6db700ad917b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:19.759334 11525 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Bootstrap starting.
I20260807 11:17:19.763417 11523 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Bootstrap starting.
I20260807 11:17:19.763824 11525 tablet_bootstrap.cc:654] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:19.765167 11524 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Bootstrap starting.
I20260807 11:17:19.765292 11525 log.cc:824] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:19.767203 11523 tablet_bootstrap.cc:654] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:19.768497 11523 log.cc:824] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:19.769629 11524 tablet_bootstrap.cc:654] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:19.771159 11524 log.cc:824] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:19.772466 11523 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: No bootstrap required, opened a new log
I20260807 11:17:19.772751 11523 ts_tablet_manager.cc:1397] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 11:17:19.774467 11524 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: No bootstrap required, opened a new log
I20260807 11:17:19.774750 11525 tablet_bootstrap.cc:492] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: No bootstrap required, opened a new log
I20260807 11:17:19.774784 11524 ts_tablet_manager.cc:1397] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 11:17:19.775031 11525 ts_tablet_manager.cc:1397] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260807 11:17:19.780658 11523 raft_consensus.cc:348] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.781136 11523 raft_consensus.cc:374] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:19.781332 11523 raft_consensus.cc:729] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Initialized, Role: FOLLOWER
I20260807 11:17:19.781991 11523 consensus_queue.cc:260] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.783695 11523 ts_tablet_manager.cc:1428] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 11:17:19.785725 11525 raft_consensus.cc:348] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.786172 11525 raft_consensus.cc:374] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:19.786266 11525 raft_consensus.cc:729] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Initialized, Role: FOLLOWER
I20260807 11:17:19.786906 11525 consensus_queue.cc:260] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.787048 11524 raft_consensus.cc:348] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.787490 11524 raft_consensus.cc:374] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:19.787642 11524 raft_consensus.cc:729] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Initialized, Role: FOLLOWER
I20260807 11:17:19.788846 11525 ts_tablet_manager.cc:1428] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 11:17:19.789399 11524 consensus_queue.cc:260] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:19.791538 11524 ts_tablet_manager.cc:1428] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260807 11:17:19.845619 11206 tablet.cc:2369] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:19.893644 11352 tablet.cc:2369] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:19.975656 11499 tablet.cc:2369] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:17:20.029812 11529 raft_consensus.cc:482] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:20.030129 11529 raft_consensus.cc:504] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:20.031981 11530 raft_consensus.cc:482] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:20.032114 11529 leader_election.cc:288] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073), 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971)
I20260807 11:17:20.032352 11530 raft_consensus.cc:504] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:20.034396 11530 leader_election.cc:288] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139), 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971)
I20260807 11:17:20.043792 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.032622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:46605 (local address 127.2.223.130:35073)
0807 11:17:20.032868 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.032877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.033875 (+ 998us) server_negotiation.cc:407] Connection header received
0807 11:17:20.034183 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.034188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.034302 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.034414 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.035647 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.037163 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.040789 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.042542 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.042701 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.042782 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.043357 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.043499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":65}
I20260807 11:17:20.044691 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.032680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.2.223.131:46605)
0807 11:17:20.033745 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.033771 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.034005 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.034637 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.034656 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.034689 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.035427 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.035445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.037411 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.037420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.040499 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.040519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.042967 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.042974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.043159 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.043193 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.044107 (+ 914us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.044286 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":592,"threads_started":1}
I20260807 11:17:20.045703 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442" is_pre_election: true
I20260807 11:17:20.046322 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.035065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:58231 (local address 127.2.223.131:41139)
0807 11:17:20.035272 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.035279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.035786 (+ 507us) server_negotiation.cc:407] Connection header received
0807 11:17:20.036103 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.036111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.036217 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.036293 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.037363 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.039007 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.043771 (+ 4764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.045220 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.045351 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.045417 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.045892 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.046033 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260807 11:17:20.046263 11303 raft_consensus.cc:2454] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 0.
I20260807 11:17:20.046464 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.034606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41139 (local address 127.2.223.130:58231)
0807 11:17:20.035679 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.035697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.035896 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.036537 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.036545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.036574 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.037160 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.037175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.039302 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.039310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.043590 (+ 4280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.043603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.045497 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.045506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.045682 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.045730 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.045844 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.046003 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":4,"thread_start_us":1063,"threads_started":1}
I20260807 11:17:20.047232 11372 leader_election.cc:302] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd1ce5ccc10439b985c56442c3ae442, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:20.047472 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e" is_pre_election: true
I20260807 11:17:20.047750 11529 raft_consensus.cc:2781] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:20.047938 11451 raft_consensus.cc:2454] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 0.
I20260807 11:17:20.048147 11529 raft_consensus.cc:482] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:20.048319 11529 raft_consensus.cc:3037] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.048683 11225 leader_election.cc:302] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd1ce5ccc10439b985c56442c3ae442, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:20.049103 11530 raft_consensus.cc:2781] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:20.049253 11530 raft_consensus.cc:482] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:20.049362 11530 raft_consensus.cc:3037] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.051121 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.035337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44971 (local address 127.2.223.130:50931)
0807 11:17:20.043207 (+ 7870us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.043287 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.043408 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.044121 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.044127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.044160 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.044496 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.044506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.045967 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.045976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.048177 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.048193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.050244 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.050251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.050448 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.050509 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.050609 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.050756 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7737,"thread_start_us":7543,"threads_started":1}
I20260807 11:17:20.051636 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.032732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:36257 (local address 127.2.223.129:44971)
0807 11:17:20.032947 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.032955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.034164 (+ 1209us) server_negotiation.cc:407] Connection header received
0807 11:17:20.034255 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.034260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.034357 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.034446 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.042351 (+ 7905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.043466 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.049056 (+ 5590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.050335 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.050473 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.050564 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.051201 (+ 637us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.051337 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:17:20.051730 11529 raft_consensus.cc:504] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:20.052110 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.035221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:50931 (local address 127.2.223.129:44971)
0807 11:17:20.042629 (+ 7408us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.042638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.043528 (+ 890us) server_negotiation.cc:407] Connection header received
0807 11:17:20.043599 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.043604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.043686 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.043861 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.044693 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.045776 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.048380 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.049798 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.050002 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.050081 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.051343 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.051486 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7176,"thread_start_us":6921,"threads_started":1}
I20260807 11:17:20.052767 11529 leader_election.cc:288] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 election: Requested vote from peers 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073), 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971)
I20260807 11:17:20.052814 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442"
I20260807 11:17:20.053176 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8" is_pre_election: true
I20260807 11:17:20.053328 11530 raft_consensus.cc:504] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:20.053653 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.033162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44971 (local address 127.2.223.131:36257)
0807 11:17:20.034006 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.034021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.034146 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.034659 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.034665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.034724 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.042012 (+ 7288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.042025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.043721 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.043726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.048780 (+ 5054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.048795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.050846 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.050852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.051018 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.051049 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.052091 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.052356 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"spinlock_wait_cycles":28032,"thread_start_us":524,"threads_started":1}
I20260807 11:17:20.053838 11303 raft_consensus.cc:2379] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3de5b4460a14551b9b216747223f98e in current term 1: Already voted for candidate 7bd1ce5ccc10439b985c56442c3ae442 in this term.
I20260807 11:17:20.053778 11157 raft_consensus.cc:2454] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 0.
I20260807 11:17:20.054138 11530 leader_election.cc:288] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 election: Requested vote from peers a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139), 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971)
I20260807 11:17:20.054636 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8"
I20260807 11:17:20.054582 11156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8" is_pre_election: true
I20260807 11:17:20.054922 11157 raft_consensus.cc:3037] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.055166 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e"
I20260807 11:17:20.055442 11451 raft_consensus.cc:2379] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bd1ce5ccc10439b985c56442c3ae442 in current term 1: Already voted for candidate a3de5b4460a14551b9b216747223f98e in this term.
I20260807 11:17:20.055538 11151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a494181193424f02a2e6db700ad917b0" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8"
I20260807 11:17:20.056205 11225 leader_election.cc:302] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bd1ce5ccc10439b985c56442c3ae442; no voters: 4d10d0a809af40e4bc146614d8b208a8, a3de5b4460a14551b9b216747223f98e
I20260807 11:17:20.056497 11530 raft_consensus.cc:2726] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:17:20.057667 11157 raft_consensus.cc:2454] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 1.
I20260807 11:17:20.058079 11375 leader_election.cc:302] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, a3de5b4460a14551b9b216747223f98e; no voters: 7bd1ce5ccc10439b985c56442c3ae442
I20260807 11:17:20.058544 11529 raft_consensus.cc:2781] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:20.059069 11529 raft_consensus.cc:686] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 LEADER]: Becoming Leader. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Running, Role: LEADER
I20260807 11:17:20.059614 11529 consensus_queue.cc:237] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } }
I20260807 11:17:20.064899 10988 catalog_manager.cc:5495] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3de5b4460a14551b9b216747223f98e (127.2.223.131). New cstate: current_term: 1 leader_uuid: "a3de5b4460a14551b9b216747223f98e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } health_report { overall_health: UNKNOWN } } }
I20260807 11:17:20.113658 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.106106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.2.223.131:41139)
0807 11:17:20.106394 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.106398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.106418 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:17:20.106728 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.106755 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.106873 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.107051 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:20.107517 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.108726 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.110774 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.111224 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.111351 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.112028 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.112964 (+ 936us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.113138 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.113209 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.113298 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260807 11:17:20.123638 11303 raft_consensus.cc:1261] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Refusing update from remote peer a3de5b4460a14551b9b216747223f98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.124559 11538 consensus_queue.cc:1035] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.124914 11157 raft_consensus.cc:1261] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Refusing update from remote peer a3de5b4460a14551b9b216747223f98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.125783 11538 consensus_queue.cc:1035] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.134505 11541 mvcc.cc:205] Tried to move back new op lower bound from 7315871498734419968 to 7315871498488086528. Current Snapshot: MvccSnapshot[applied={T|T < 7315871498734419968}]
I20260807 11:17:20.136169 11542 mvcc.cc:205] Tried to move back new op lower bound from 7315871498734419968 to 7315871498488086528. Current Snapshot: MvccSnapshot[applied={T|T < 7315871498734419968}]
I20260807 11:17:20.149360 10988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.149698 10988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.153602 10988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Sending DeleteTablet for 3 replicas of tablet a494181193424f02a2e6db700ad917b0
I20260807 11:17:20.154891 11133 tablet_service.cc:1514] Processing DeleteTablet for tablet a494181193424f02a2e6db700ad917b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:57158
I20260807 11:17:20.155233 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet a494181193424f02a2e6db700ad917b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:47238
I20260807 11:17:20.155223 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet a494181193424f02a2e6db700ad917b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60114
I20260807 11:17:20.157541 10988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:20.158335 10988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:20.167594 11552 tablet_replica.cc:330] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: stopping tablet replica
I20260807 11:17:20.167675 11276 tablet_service.cc:1467] Processing CreateTablet for tablet 777b68321c45423598d586cd14979f23 (DEFAULT_TABLE table=test-table [id=58dece1d720b40d784f51513f723c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.168351 11276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777b68321c45423598d586cd14979f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.168365 11424 tablet_service.cc:1467] Processing CreateTablet for tablet 777b68321c45423598d586cd14979f23 (DEFAULT_TABLE table=test-table [id=58dece1d720b40d784f51513f723c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.168395 11552 raft_consensus.cc:2229] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:20.168962 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777b68321c45423598d586cd14979f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.169469 11552 raft_consensus.cc:2258] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.171974 11552 ts_tablet_manager.cc:1905] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.175108 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 777b68321c45423598d586cd14979f23 (DEFAULT_TABLE table=test-table [id=58dece1d720b40d784f51513f723c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.175796 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777b68321c45423598d586cd14979f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.177692 11550 tablet_replica.cc:330] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: stopping tablet replica
I20260807 11:17:20.178287 11551 tablet_replica.cc:330] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: stopping tablet replica
I20260807 11:17:20.179174 11523 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Bootstrap starting.
I20260807 11:17:20.180106 11551 raft_consensus.cc:2229] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.180687 11551 raft_consensus.cc:2258] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.180773 11523 tablet_bootstrap.cc:654] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.181630 11550 raft_consensus.cc:2229] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.182113 11550 raft_consensus.cc:2258] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.182868 11524 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Bootstrap starting.
I20260807 11:17:20.184202 11525 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Bootstrap starting.
I20260807 11:17:20.184549 11524 tablet_bootstrap.cc:654] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.185945 11525 tablet_bootstrap.cc:654] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.186175 11523 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: No bootstrap required, opened a new log
I20260807 11:17:20.186362 11523 ts_tablet_manager.cc:1397] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 11:17:20.187603 11523 raft_consensus.cc:348] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.187857 11523 raft_consensus.cc:374] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.187981 11523 raft_consensus.cc:729] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.188324 11523 consensus_queue.cc:260] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.189168 11523 ts_tablet_manager.cc:1428] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:20.196089 11550 ts_tablet_manager.cc:1905] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.197135 11552 ts_tablet_manager.cc:1918] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.197336 11552 log.cc:1196] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/a494181193424f02a2e6db700ad917b0
I20260807 11:17:20.197947 11552 ts_tablet_manager.cc:1939] T a494181193424f02a2e6db700ad917b0 P a3de5b4460a14551b9b216747223f98e: Deleting consensus metadata
I20260807 11:17:20.199903 11551 ts_tablet_manager.cc:1905] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.199993 10972 catalog_manager.cc:4841] TS a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139): tablet a494181193424f02a2e6db700ad917b0 (table test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]) successfully deleted
I20260807 11:17:20.206274 11524 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: No bootstrap required, opened a new log
I20260807 11:17:20.206509 11524 ts_tablet_manager.cc:1397] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20260807 11:17:20.207655 11524 raft_consensus.cc:348] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.207907 11524 raft_consensus.cc:374] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.208009 11524 raft_consensus.cc:729] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.208382 11524 consensus_queue.cc:260] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.209161 11524 ts_tablet_manager.cc:1428] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 11:17:20.212646 11551 ts_tablet_manager.cc:1918] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.212833 11551 log.cc:1196] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/a494181193424f02a2e6db700ad917b0
I20260807 11:17:20.213272 11550 ts_tablet_manager.cc:1918] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.213480 11551 ts_tablet_manager.cc:1939] T a494181193424f02a2e6db700ad917b0 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting consensus metadata
I20260807 11:17:20.213461 11550 log.cc:1196] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/a494181193424f02a2e6db700ad917b0
I20260807 11:17:20.214105 11550 ts_tablet_manager.cc:1939] T a494181193424f02a2e6db700ad917b0 P 4d10d0a809af40e4bc146614d8b208a8: Deleting consensus metadata
I20260807 11:17:20.215950 10975 catalog_manager.cc:4841] TS 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971): tablet a494181193424f02a2e6db700ad917b0 (table test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]) successfully deleted
I20260807 11:17:20.215966 10972 catalog_manager.cc:4841] TS 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073): tablet a494181193424f02a2e6db700ad917b0 (table test-table [id=93cb7dd4c59c4b9cb850e712742fe6cc]) successfully deleted
I20260807 11:17:20.217762 11525 tablet_bootstrap.cc:492] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: No bootstrap required, opened a new log
I20260807 11:17:20.217936 11525 ts_tablet_manager.cc:1397] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.004s
I20260807 11:17:20.218910 11525 raft_consensus.cc:348] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.219167 11525 raft_consensus.cc:374] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.219290 11525 raft_consensus.cc:729] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.219565 11525 consensus_queue.cc:260] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.220332 11525 ts_tablet_manager.cc:1428] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 11:17:20.238636 11538 raft_consensus.cc:482] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:20.238883 11538 raft_consensus.cc:504] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.239790 11538 leader_election.cc:288] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971), 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073)
I20260807 11:17:20.240406 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777b68321c45423598d586cd14979f23" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442" is_pre_election: true
I20260807 11:17:20.240419 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777b68321c45423598d586cd14979f23" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8" is_pre_election: true
I20260807 11:17:20.240705 11157 raft_consensus.cc:2454] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 0.
I20260807 11:17:20.240736 11303 raft_consensus.cc:2454] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 0.
I20260807 11:17:20.241200 11372 leader_election.cc:302] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd1ce5ccc10439b985c56442c3ae442, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:20.241500 11538 raft_consensus.cc:2781] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:20.241667 11538 raft_consensus.cc:482] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:20.241788 11538 raft_consensus.cc:3037] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.244537 11538 raft_consensus.cc:504] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.245236 11538 leader_election.cc:288] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 election: Requested vote from peers 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971), 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073)
I20260807 11:17:20.245776 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777b68321c45423598d586cd14979f23" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8"
I20260807 11:17:20.245806 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777b68321c45423598d586cd14979f23" candidate_uuid: "a3de5b4460a14551b9b216747223f98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442"
I20260807 11:17:20.246031 11157 raft_consensus.cc:3037] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.246062 11303 raft_consensus.cc:3037] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.249188 11303 raft_consensus.cc:2454] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 1.
I20260807 11:17:20.249199 11157 raft_consensus.cc:2454] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3de5b4460a14551b9b216747223f98e in term 1.
I20260807 11:17:20.249737 11372 leader_election.cc:302] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, 7bd1ce5ccc10439b985c56442c3ae442, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:20.250151 11538 raft_consensus.cc:2781] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:20.250341 11538 raft_consensus.cc:686] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 1 LEADER]: Becoming Leader. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Running, Role: LEADER
I20260807 11:17:20.250727 11538 consensus_queue.cc:237] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.253754 10987 catalog_manager.cc:5495] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3de5b4460a14551b9b216747223f98e (127.2.223.131). New cstate: current_term: 1 leader_uuid: "a3de5b4460a14551b9b216747223f98e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } health_report { overall_health: HEALTHY } } }
Aug 07 11:17:20 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101436, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.190@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:20.316349 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.301213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.2.223.190:37227)
0807 11:17:20.302259 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.302266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.302293 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:20.302333 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.302336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.302388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.302526 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:20.303643 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.304427 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.305846 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.306254 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.306398 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.310537 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:20.310546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:20.310553 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:20.310592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:20.313449 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:20.314251 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:20.314259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:20.314262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:20.314393 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:20.314792 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:20.314799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:20.314805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:20.315072 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:20.315250 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.315640 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.315978 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":906,"thread_start_us":735,"threads_started":1}
I20260807 11:17:20.316346 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.301125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:37227 (local address 127.0.0.1:51088)
0807 11:17:20.301926 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.301953 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.302148 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.302563 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.302569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.303285 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:20.303478 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.303488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.304570 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.304577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.305699 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.305713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.306423 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.306429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.306658 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.307463 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:20.307481 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:20.310250 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:20.313613 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:20.313630 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:20.313639 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:20.314126 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:20.314533 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:20.314539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:20.314542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:20.314685 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:20.315210 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:20.315219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:20.315380 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.315548 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.315844 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":424,"threads_started":1}
I20260807 11:17:20.322086 10987 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:51088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:17:20.323159 10987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:20.330644 11132 tablet_service.cc:1467] Processing CreateTablet for tablet b273a35c6cf443358a8a851b09030864 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1170e4ab2bec4fe0bedd26dd2e23714b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:17:20.330811 11424 tablet_service.cc:1467] Processing CreateTablet for tablet b273a35c6cf443358a8a851b09030864 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1170e4ab2bec4fe0bedd26dd2e23714b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:17:20.331264 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b273a35c6cf443358a8a851b09030864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.331413 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b273a35c6cf443358a8a851b09030864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.335031 11279 tablet_service.cc:1467] Processing CreateTablet for tablet b273a35c6cf443358a8a851b09030864 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1170e4ab2bec4fe0bedd26dd2e23714b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:17:20.335602 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b273a35c6cf443358a8a851b09030864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.338387 11524 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: Bootstrap starting.
I20260807 11:17:20.339087 11523 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: Bootstrap starting.
I20260807 11:17:20.340234 11524 tablet_bootstrap.cc:654] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.341187 11523 tablet_bootstrap.cc:654] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.349840 11523 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: No bootstrap required, opened a new log
I20260807 11:17:20.350028 11523 ts_tablet_manager.cc:1397] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260807 11:17:20.350860 11523 raft_consensus.cc:348] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.351239 11523 raft_consensus.cc:374] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.351339 11523 raft_consensus.cc:729] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.351593 11523 consensus_queue.cc:260] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.352068 11538 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.352412 11538 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.353144 11523 ts_tablet_manager.cc:1428] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:20.354413 11524 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: No bootstrap required, opened a new log
I20260807 11:17:20.354625 11524 ts_tablet_manager.cc:1397] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20260807 11:17:20.354877 11525 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: Bootstrap starting.
I20260807 11:17:20.355373 11524 raft_consensus.cc:348] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.355618 11524 raft_consensus.cc:374] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.355687 11524 raft_consensus.cc:729] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.355950 11524 consensus_queue.cc:260] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.356395 11531 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.356721 11525 tablet_bootstrap.cc:654] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.356753 11531 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.358096 11524 ts_tablet_manager.cc:1428] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:20.364049 11525 tablet_bootstrap.cc:492] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: No bootstrap required, opened a new log
I20260807 11:17:20.364310 11525 ts_tablet_manager.cc:1397] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260807 11:17:20.365131 11525 raft_consensus.cc:348] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.365434 11525 raft_consensus.cc:374] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.365556 11525 raft_consensus.cc:729] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.365870 11525 consensus_queue.cc:260] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.366333 11530 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.366672 11530 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.369098 11525 ts_tablet_manager.cc:1428] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260807 11:17:20.372028 11530 raft_consensus.cc:482] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:20.372177 11530 raft_consensus.cc:504] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.372730 11530 leader_election.cc:288] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971), a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139)
I20260807 11:17:20.373306 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b273a35c6cf443358a8a851b09030864" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e" is_pre_election: true
I20260807 11:17:20.373347 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b273a35c6cf443358a8a851b09030864" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8" is_pre_election: true
I20260807 11:17:20.373580 11451 raft_consensus.cc:2454] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 0.
I20260807 11:17:20.373595 11157 raft_consensus.cc:2454] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 0.
I20260807 11:17:20.374023 11228 leader_election.cc:302] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, 7bd1ce5ccc10439b985c56442c3ae442; no voters:
I20260807 11:17:20.374631 11530 raft_consensus.cc:2781] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:20.374719 11530 raft_consensus.cc:482] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:20.374805 11530 raft_consensus.cc:3037] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.377684 11530 raft_consensus.cc:504] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.378634 11157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b273a35c6cf443358a8a851b09030864" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d10d0a809af40e4bc146614d8b208a8"
I20260807 11:17:20.378851 11157 raft_consensus.cc:3037] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.379401 11530 leader_election.cc:288] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 election: Requested vote from peers 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971), a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139)
I20260807 11:17:20.380049 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b273a35c6cf443358a8a851b09030864" candidate_uuid: "7bd1ce5ccc10439b985c56442c3ae442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e"
I20260807 11:17:20.380264 11451 raft_consensus.cc:3037] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.381850 11157 raft_consensus.cc:2454] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 1.
I20260807 11:17:20.382221 11228 leader_election.cc:302] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, 7bd1ce5ccc10439b985c56442c3ae442; no voters:
I20260807 11:17:20.382553 11530 raft_consensus.cc:2781] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:20.383097 11530 raft_consensus.cc:686] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Running, Role: LEADER
I20260807 11:17:20.383518 11451 raft_consensus.cc:2454] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bd1ce5ccc10439b985c56442c3ae442 in term 1.
I20260807 11:17:20.383479 11530 consensus_queue.cc:237] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.385730 11568 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: TxnStatusTablet state changed. Reason: New leader 7bd1ce5ccc10439b985c56442c3ae442. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.386101 11568 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:17:20.387696 10987 catalog_manager.cc:5495] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260807 11:17:20.387933 11570 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:17:20.388164 11570 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:17:20.410554 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.403064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.0.0.1:47250)
0807 11:17:20.403467 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.403488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.403653 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.404142 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.404147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.404159 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:17:20.404364 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.404375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.405847 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.405853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.407578 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.407594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.408643 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.408650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.408890 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.408968 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.409933 (+ 965us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.409955 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.410047 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.410217 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260807 11:17:20.410555 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.402985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.2.223.130:35073)
0807 11:17:20.403147 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.403153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.403574 (+ 421us) server_negotiation.cc:407] Connection header received
0807 11:17:20.403860 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.403865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.403956 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.404093 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:20.404519 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.405686 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.407758 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.408460 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.408591 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.409199 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.409807 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.409949 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.410102 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.410187 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 11:17:20.415283 11157 raft_consensus.cc:1261] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Refusing update from remote peer 7bd1ce5ccc10439b985c56442c3ae442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.415387 11451 raft_consensus.cc:1261] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Refusing update from remote peer 7bd1ce5ccc10439b985c56442c3ae442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.416081 11530 consensus_queue.cc:1035] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.416497 11568 consensus_queue.cc:1035] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.431530 11531 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: TxnStatusTablet state changed. Reason: New leader 7bd1ce5ccc10439b985c56442c3ae442. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.431869 11531 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.435403 11538 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: TxnStatusTablet state changed. Reason: New leader 7bd1ce5ccc10439b985c56442c3ae442. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.435751 11538 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.436906 11544 mvcc.cc:205] Tried to move back new op lower bound from 7315871499932467200 to 7315871499814957056. Current Snapshot: MvccSnapshot[applied={T|T < 7315871499932467200}]
I20260807 11:17:20.437487 11530 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.437605 11531 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.437836 11530 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:17:20.437851 11531 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 4d10d0a809af40e4bc146614d8b208a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.444516 11568 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.445266 11568 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P 7bd1ce5ccc10439b985c56442c3ae442: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:17:20.446712 11538 tablet_replica.cc:439] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bd1ce5ccc10439b985c56442c3ae442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } }
I20260807 11:17:20.447010 11538 tablet_replica.cc:442] T b273a35c6cf443358a8a851b09030864 P a3de5b4460a14551b9b216747223f98e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:17:20.469141 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.461001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:40647 (local address 127.2.223.130:35073)
0807 11:17:20.461221 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.461228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.462051 (+ 823us) server_negotiation.cc:407] Connection header received
0807 11:17:20.462268 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.462275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.462364 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.462467 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:20.463133 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.464354 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.466730 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.467470 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.467606 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.468226 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.468366 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.468462 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.468648 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.468739 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:17:20.469290 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.461074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.2.223.131:40647)
0807 11:17:20.461909 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.461935 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.462116 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.462535 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.462540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.462554 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:17:20.462943 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.462957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.464533 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.464540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.466575 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.466595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.467655 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.467662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.467855 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.467934 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.468486 (+ 552us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.468498 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.468637 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.468794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":469,"threads_started":1}
I20260807 11:17:20.481626 11157 raft_consensus.cc:1261] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Refusing update from remote peer a3de5b4460a14551b9b216747223f98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.481843 11303 raft_consensus.cc:1261] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Refusing update from remote peer a3de5b4460a14551b9b216747223f98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.482462 11529 consensus_queue.cc:1035] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.482992 11529 consensus_queue.cc:1035] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.563815 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.542747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:47881 (local address 127.2.223.131:41139)
0807 11:17:20.542971 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.542977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.555141 (+ 12164us) server_negotiation.cc:407] Connection header received
0807 11:17:20.555344 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.555353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.555445 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.555547 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:20.556012 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.557166 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.561444 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.562214 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.562360 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.562924 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.563082 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.563188 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.563396 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.563482 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:17:20.566428 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.542318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41139 (local address 127.2.223.130:47881)
0807 11:17:20.554993 (+ 12675us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.555014 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.555194 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.555588 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.555593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.555607 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:17:20.555828 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.555838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.557381 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.557387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.561268 (+ 3881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.561284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.562401 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.562413 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.562629 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.562711 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.563222 (+ 511us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.563234 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.563311 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.563472 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12464,"thread_start_us":12348,"threads_started":1}
I20260807 11:17:20.628649 10987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.628980 10987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.632890 10987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Sending DeleteTablet for 3 replicas of tablet 777b68321c45423598d586cd14979f23
I20260807 11:17:20.633946 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 777b68321c45423598d586cd14979f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:57158
I20260807 11:17:20.633958 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet 777b68321c45423598d586cd14979f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60114
I20260807 11:17:20.634285 11552 tablet_replica.cc:330] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: stopping tablet replica
I20260807 11:17:20.634274 11550 tablet_replica.cc:330] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: stopping tablet replica
I20260807 11:17:20.634464 11552 raft_consensus.cc:2229] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:20.634506 11550 raft_consensus.cc:2229] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.634610 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet 777b68321c45423598d586cd14979f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:47238
I20260807 11:17:20.634738 11550 raft_consensus.cc:2258] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.634856 11552 raft_consensus.cc:2258] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.635174 11551 tablet_replica.cc:330] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: stopping tablet replica
I20260807 11:17:20.635426 11551 raft_consensus.cc:2229] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.635648 11551 raft_consensus.cc:2258] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.637257 11551 ts_tablet_manager.cc:1905] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.640976 11550 ts_tablet_manager.cc:1905] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.647382 11552 ts_tablet_manager.cc:1905] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.656906 11550 ts_tablet_manager.cc:1918] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:20.657086 11550 log.cc:1196] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/777b68321c45423598d586cd14979f23
I20260807 11:17:20.657552 11550 ts_tablet_manager.cc:1939] T 777b68321c45423598d586cd14979f23 P 4d10d0a809af40e4bc146614d8b208a8: Deleting consensus metadata
I20260807 11:17:20.658041 11552 ts_tablet_manager.cc:1918] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:20.658856 10975 catalog_manager.cc:4841] TS 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971): tablet 777b68321c45423598d586cd14979f23 (table test-table [id=58dece1d720b40d784f51513f723c043]) successfully deleted
I20260807 11:17:20.658955 11551 ts_tablet_manager.cc:1918] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:20.659134 11551 log.cc:1196] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/777b68321c45423598d586cd14979f23
I20260807 11:17:20.659264 11552 log.cc:1196] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/777b68321c45423598d586cd14979f23
I20260807 11:17:20.659674 11551 ts_tablet_manager.cc:1939] T 777b68321c45423598d586cd14979f23 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting consensus metadata
I20260807 11:17:20.659718 11552 ts_tablet_manager.cc:1939] T 777b68321c45423598d586cd14979f23 P a3de5b4460a14551b9b216747223f98e: Deleting consensus metadata
I20260807 11:17:20.660835 10972 catalog_manager.cc:4841] TS 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073): tablet 777b68321c45423598d586cd14979f23 (table test-table [id=58dece1d720b40d784f51513f723c043]) successfully deleted
I20260807 11:17:20.661863 10972 catalog_manager.cc:4841] TS a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139): tablet 777b68321c45423598d586cd14979f23 (table test-table [id=58dece1d720b40d784f51513f723c043]) successfully deleted
Aug 07 11:17:20 dist-test-slave-5l3k krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:17:20 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:20 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:17:20.740783 11563 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:20.725946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.2.223.190:37227)
0807 11:17:20.726138 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.726146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.726742 (+ 596us) server_negotiation.cc:407] Connection header received
0807 11:17:20.726963 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.726969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.727058 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.727171 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:20.729165 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.730331 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.732312 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.732992 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.733137 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.740490 (+ 7353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51098: BlockingRecv error: recv got EOF from 127.0.0.1:51098 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 11:17:20 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM
I20260807 11:17:20.776716 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.758614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.2.223.190:37227)
0807 11:17:20.758853 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.758860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.759010 (+ 150us) server_negotiation.cc:407] Connection header received
0807 11:17:20.759313 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.759320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.759418 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.759542 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:20.761164 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.762377 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.764080 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.764807 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.764943 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.770172 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:20.770189 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:20.770194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:20.770244 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:20.773838 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:20.774579 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:20.774586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:20.774589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:20.774767 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:20.775205 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:20.775211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:20.775216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:20.775574 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:20.775857 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.776061 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.776321 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:17:20.780853 10987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:20.781618 10987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:20.788900 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 (DEFAULT_TABLE table=test-table [id=c37a42743ebf4c56b3b21aecca64d820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.789430 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4018238a3e924e4a9afcf71b2dd57394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.789667 11279 tablet_service.cc:1467] Processing CreateTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 (DEFAULT_TABLE table=test-table [id=c37a42743ebf4c56b3b21aecca64d820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.789688 11424 tablet_service.cc:1467] Processing CreateTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 (DEFAULT_TABLE table=test-table [id=c37a42743ebf4c56b3b21aecca64d820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.790126 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4018238a3e924e4a9afcf71b2dd57394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.790136 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4018238a3e924e4a9afcf71b2dd57394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.796686 11524 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Bootstrap starting.
I20260807 11:17:20.797236 11525 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Bootstrap starting.
I20260807 11:17:20.798373 11523 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Bootstrap starting.
I20260807 11:17:20.798575 11524 tablet_bootstrap.cc:654] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.799312 11525 tablet_bootstrap.cc:654] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.800176 11523 tablet_bootstrap.cc:654] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.809530 11524 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: No bootstrap required, opened a new log
I20260807 11:17:20.809808 11524 ts_tablet_manager.cc:1397] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260807 11:17:20.809978 11525 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: No bootstrap required, opened a new log
I20260807 11:17:20.810153 11525 ts_tablet_manager.cc:1397] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260807 11:17:20.810158 11523 tablet_bootstrap.cc:492] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: No bootstrap required, opened a new log
I20260807 11:17:20.810318 11523 ts_tablet_manager.cc:1397] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 11:17:20.811031 11524 raft_consensus.cc:348] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.811069 11525 raft_consensus.cc:348] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.811134 11523 raft_consensus.cc:348] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.811374 11523 raft_consensus.cc:374] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.811390 11525 raft_consensus.cc:374] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.811478 11523 raft_consensus.cc:729] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.811499 11525 raft_consensus.cc:729] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.811515 11524 raft_consensus.cc:374] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.811731 11524 raft_consensus.cc:729] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.811789 11523 consensus_queue.cc:260] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.811800 11525 consensus_queue.cc:260] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.812201 11524 consensus_queue.cc:260] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.813552 11524 ts_tablet_manager.cc:1428] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 11:17:20.814334 11523 ts_tablet_manager.cc:1428] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 11:17:20.815153 11525 ts_tablet_manager.cc:1428] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:17:20.849565 11531 raft_consensus.cc:482] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:20.849872 11531 raft_consensus.cc:504] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.852173 11531 leader_election.cc:288] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073), a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139)
I20260807 11:17:20.863629 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.854281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:46853 (local address 127.2.223.131:41139)
0807 11:17:20.854505 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.854514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.856728 (+ 2214us) server_negotiation.cc:407] Connection header received
0807 11:17:20.856877 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.856882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.856959 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.857058 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.857800 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.858896 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.861200 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.862339 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.862518 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.862597 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.863129 (+ 532us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.863280 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:17:20.864725 11602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.853868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:41139 (local address 127.2.223.129:46853)
0807 11:17:20.856570 (+ 2702us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.856594 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.856741 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.857221 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.857227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.857252 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.857522 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.857532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.859129 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.859134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.860997 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.861012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.862737 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.862743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.862924 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.862957 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.864182 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.864344 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"thread_start_us":2460,"threads_started":1}
I20260807 11:17:20.867218 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4018238a3e924e4a9afcf71b2dd57394" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e" is_pre_election: true
I20260807 11:17:20.867511 11451 raft_consensus.cc:2454] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 0.
I20260807 11:17:20.868183 11080 leader_election.cc:302] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:20.868716 11531 raft_consensus.cc:2781] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:20.868851 11531 raft_consensus.cc:482] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:20.868937 11531 raft_consensus.cc:3037] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.872190 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.852879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:47675 (local address 127.2.223.130:35073)
0807 11:17:20.853205 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.853215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.854021 (+ 806us) server_negotiation.cc:407] Connection header received
0807 11:17:20.854571 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.854612 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.854794 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.854927 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:20.865584 (+ 10657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.866457 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.869217 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.871031 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.871241 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.871342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.871693 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.871862 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:17:20.872218 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.852711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.2.223.129:47675)
0807 11:17:20.853878 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:20.853896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:20.854223 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:20.855388 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:20.855397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:20.855464 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:20.857192 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.857223 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.866673 (+ 9450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.866681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.868744 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:20.868765 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.871259 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.871267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:20.871484 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.871527 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 11:17:20.871662 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 11:17:20.871823 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":745,"threads_started":1}
I20260807 11:17:20.872438 11531 raft_consensus.cc:504] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.873056 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4018238a3e924e4a9afcf71b2dd57394" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442" is_pre_election: true
I20260807 11:17:20.873237 11531 leader_election.cc:288] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 election: Requested vote from peers 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073), a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139)
I20260807 11:17:20.873379 11302 raft_consensus.cc:2454] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 0.
I20260807 11:17:20.873759 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4018238a3e924e4a9afcf71b2dd57394" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442"
I20260807 11:17:20.874010 11303 raft_consensus.cc:3037] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.875361 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4018238a3e924e4a9afcf71b2dd57394" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e"
I20260807 11:17:20.875595 11451 raft_consensus.cc:3037] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:20.877312 11303 raft_consensus.cc:2454] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 1.
I20260807 11:17:20.877789 11080 leader_election.cc:302] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, 7bd1ce5ccc10439b985c56442c3ae442; no voters:
I20260807 11:17:20.878139 11531 raft_consensus.cc:2781] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:20.878691 11451 raft_consensus.cc:2454] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 1.
I20260807 11:17:20.878968 11531 raft_consensus.cc:686] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Running, Role: LEADER
I20260807 11:17:20.879664 11531 consensus_queue.cc:237] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } }
I20260807 11:17:20.883775 10985 catalog_manager.cc:5495] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129). New cstate: current_term: 1 leader_uuid: "4d10d0a809af40e4bc146614d8b208a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260807 11:17:20.924568 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:20.915783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.2.223.129:44971)
0807 11:17:20.919001 (+ 3218us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:20.919005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:20.919036 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:17:20.919192 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:20.919197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:20.919286 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:20.919533 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:20.919969 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.920853 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.922202 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:20.922698 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:20.922833 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:20.923315 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:20.923890 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:20.924055 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:20.924153 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 11:17:20.924235 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3088,"thread_start_us":2933,"threads_started":1}
I20260807 11:17:20.928543 11451 raft_consensus.cc:1261] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Refusing update from remote peer 4d10d0a809af40e4bc146614d8b208a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.928579 11303 raft_consensus.cc:1261] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Refusing update from remote peer 4d10d0a809af40e4bc146614d8b208a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:20.929337 11603 consensus_queue.cc:1035] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.929728 11531 consensus_queue.cc:1035] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:20.944782 10987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.945032 10987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:20.948462 10987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Sending DeleteTablet for 3 replicas of tablet 4018238a3e924e4a9afcf71b2dd57394
I20260807 11:17:20.949156 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:57158
I20260807 11:17:20.949563 11550 tablet_replica.cc:330] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: stopping tablet replica
I20260807 11:17:20.949764 11550 raft_consensus.cc:2229] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:20.950374 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:47238
I20260807 11:17:20.950634 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60114
I20260807 11:17:20.950943 11551 tablet_replica.cc:330] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: stopping tablet replica
I20260807 11:17:20.951148 11551 raft_consensus.cc:2229] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.951416 11551 raft_consensus.cc:2258] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.951767 11552 tablet_replica.cc:330] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: stopping tablet replica
I20260807 11:17:20.951967 11552 raft_consensus.cc:2229] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:20.952203 11552 raft_consensus.cc:2258] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.952504 10987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 11:17:20.952847 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4018238a3e924e4a9afcf71b2dd57394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:20 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60114
W20260807 11:17:20.953244 10987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:20.953487 11551 ts_tablet_manager.cc:1905] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.956117 11552 ts_tablet_manager.cc:1905] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.960071 11551 ts_tablet_manager.cc:1918] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.960228 11551 log.cc:1196] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/4018238a3e924e4a9afcf71b2dd57394
I20260807 11:17:20.960675 11551 ts_tablet_manager.cc:1939] T 4018238a3e924e4a9afcf71b2dd57394 P 7bd1ce5ccc10439b985c56442c3ae442: Deleting consensus metadata
I20260807 11:17:20.961786 10972 catalog_manager.cc:4841] TS 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073): tablet 4018238a3e924e4a9afcf71b2dd57394 (table test-table [id=c37a42743ebf4c56b3b21aecca64d820]) successfully deleted
I20260807 11:17:20.963657 11552 ts_tablet_manager.cc:1918] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.964179 11552 log.cc:1196] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/4018238a3e924e4a9afcf71b2dd57394
I20260807 11:17:20.964707 11552 ts_tablet_manager.cc:1939] T 4018238a3e924e4a9afcf71b2dd57394 P a3de5b4460a14551b9b216747223f98e: Deleting consensus metadata
I20260807 11:17:20.966529 10972 catalog_manager.cc:4841] TS a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139): tablet 4018238a3e924e4a9afcf71b2dd57394 (table test-table [id=c37a42743ebf4c56b3b21aecca64d820]) successfully deleted
I20260807 11:17:20.966609 11424 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d112cd746c7984eff41a9132b8d (DEFAULT_TABLE table=test-table [id=5f290685230a428aba0d476d8fea163e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.967036 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d112cd746c7984eff41a9132b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.967772 11279 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d112cd746c7984eff41a9132b8d (DEFAULT_TABLE table=test-table [id=5f290685230a428aba0d476d8fea163e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:20.968251 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d112cd746c7984eff41a9132b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.972832 11550 raft_consensus.cc:2258] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:20.974002 11550 ts_tablet_manager.cc:1905] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:20.974038 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d112cd746c7984eff41a9132b8d (DEFAULT_TABLE table=test-table [id=5f290685230a428aba0d476d8fea163e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260807 11:17:20.974184 10972 catalog_manager.cc:4808] TS a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139): delete failed for tablet 4018238a3e924e4a9afcf71b2dd57394 because the tablet was not found. No further retry: Not found: Tablet not found: 4018238a3e924e4a9afcf71b2dd57394
I20260807 11:17:20.974431 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d112cd746c7984eff41a9132b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:20.974785 11525 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Bootstrap starting.
I20260807 11:17:20.976069 11525 tablet_bootstrap.cc:654] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.978081 11525 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: No bootstrap required, opened a new log
I20260807 11:17:20.978292 11525 ts_tablet_manager.cc:1397] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 11:17:20.979082 11525 raft_consensus.cc:348] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.979368 11525 raft_consensus.cc:374] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.979447 11525 raft_consensus.cc:729] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1ce5ccc10439b985c56442c3ae442, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.979787 11525 consensus_queue.cc:260] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.981038 11525 ts_tablet_manager.cc:1428] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 11:17:20.981046 11550 ts_tablet_manager.cc:1918] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:20.981184 11550 log.cc:1196] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/4018238a3e924e4a9afcf71b2dd57394
I20260807 11:17:20.981434 11523 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Bootstrap starting.
I20260807 11:17:20.981673 11550 ts_tablet_manager.cc:1939] T 4018238a3e924e4a9afcf71b2dd57394 P 4d10d0a809af40e4bc146614d8b208a8: Deleting consensus metadata
I20260807 11:17:20.982796 11524 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Bootstrap starting.
I20260807 11:17:20.983140 11523 tablet_bootstrap.cc:654] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.984735 11524 tablet_bootstrap.cc:654] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:20.986006 11523 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: No bootstrap required, opened a new log
I20260807 11:17:20.986161 11523 ts_tablet_manager.cc:1397] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:17:20.986990 11523 raft_consensus.cc:348] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.987267 11523 raft_consensus.cc:374] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.987349 11523 raft_consensus.cc:729] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3de5b4460a14551b9b216747223f98e, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.987473 10975 catalog_manager.cc:4841] TS 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971): tablet 4018238a3e924e4a9afcf71b2dd57394 (table test-table [id=c37a42743ebf4c56b3b21aecca64d820]) successfully deleted
I20260807 11:17:20.987659 11523 consensus_queue.cc:260] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.988926 11524 tablet_bootstrap.cc:492] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: No bootstrap required, opened a new log
I20260807 11:17:20.989074 11523 ts_tablet_manager.cc:1428] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:20.989110 11524 ts_tablet_manager.cc:1397] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 11:17:20.989773 11524 raft_consensus.cc:348] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.990060 11524 raft_consensus.cc:374] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:20.990150 11524 raft_consensus.cc:729] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Initialized, Role: FOLLOWER
I20260807 11:17:20.990408 11524 consensus_queue.cc:260] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:20.991093 11524 ts_tablet_manager.cc:1428] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 11:17:21.061806 11531 raft_consensus.cc:482] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:21.062048 11531 raft_consensus.cc:504] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:21.062815 11531 leader_election.cc:288] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139), 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073)
I20260807 11:17:21.063387 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d112cd746c7984eff41a9132b8d" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442" is_pre_election: true
I20260807 11:17:21.063426 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d112cd746c7984eff41a9132b8d" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e" is_pre_election: true
I20260807 11:17:21.063673 11303 raft_consensus.cc:2454] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 0.
I20260807 11:17:21.063697 11451 raft_consensus.cc:2454] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 0.
I20260807 11:17:21.064252 11080 leader_election.cc:302] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, 7bd1ce5ccc10439b985c56442c3ae442; no voters:
I20260807 11:17:21.064548 11531 raft_consensus.cc:2781] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:21.064680 11531 raft_consensus.cc:482] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:21.064801 11531 raft_consensus.cc:3037] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:21.067480 11531 raft_consensus.cc:504] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:21.068104 11531 leader_election.cc:288] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 election: Requested vote from peers a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139), 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073)
I20260807 11:17:21.068588 11451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d112cd746c7984eff41a9132b8d" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3de5b4460a14551b9b216747223f98e"
I20260807 11:17:21.068704 11303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d112cd746c7984eff41a9132b8d" candidate_uuid: "4d10d0a809af40e4bc146614d8b208a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd1ce5ccc10439b985c56442c3ae442"
I20260807 11:17:21.068802 11451 raft_consensus.cc:3037] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:21.068943 11303 raft_consensus.cc:3037] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:21.071328 11451 raft_consensus.cc:2454] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 1.
I20260807 11:17:21.071738 11080 leader_election.cc:302] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d10d0a809af40e4bc146614d8b208a8, a3de5b4460a14551b9b216747223f98e; no voters:
I20260807 11:17:21.072018 11531 raft_consensus.cc:2781] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:21.072139 11303 raft_consensus.cc:2454] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d10d0a809af40e4bc146614d8b208a8 in term 1.
I20260807 11:17:21.072201 11531 raft_consensus.cc:686] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4d10d0a809af40e4bc146614d8b208a8, State: Running, Role: LEADER
I20260807 11:17:21.072770 11531 consensus_queue.cc:237] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } }
I20260807 11:17:21.075526 10985 catalog_manager.cc:5495] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129). New cstate: current_term: 1 leader_uuid: "4d10d0a809af40e4bc146614d8b208a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d10d0a809af40e4bc146614d8b208a8" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 } health_report { overall_health: UNKNOWN } } }
I20260807 11:17:21.108047 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.102001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35073 (local address 127.2.223.129:58859)
0807 11:17:21.102705 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:21.102722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:21.102921 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:21.103263 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:21.103268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:21.103279 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:17:21.103675 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:21.103686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.104718 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.104724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:21.105943 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:21.105957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.106781 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.106791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:21.106928 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.106991 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:17:21.107507 (+ 516us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:17:21.107516 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 11:17:21.107580 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 11:17:21.107724 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":330,"threads_started":1}
I20260807 11:17:21.108052 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.101989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:58859 (local address 127.2.223.130:35073)
0807 11:17:21.102164 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:21.102169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:21.102833 (+ 664us) server_negotiation.cc:407] Connection header received
0807 11:17:21.103067 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:21.103079 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:21.103148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:21.103244 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:21.103861 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.104573 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.106095 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.106566 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.106706 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.107243 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:21.107410 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:21.107505 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:21.107633 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 11:17:21.107742 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:17:21.117691 11451 raft_consensus.cc:1261] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Refusing update from remote peer 4d10d0a809af40e4bc146614d8b208a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:21.117717 11303 raft_consensus.cc:1261] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Refusing update from remote peer 4d10d0a809af40e4bc146614d8b208a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:21.118259 11610 consensus_queue.cc:1035] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd1ce5ccc10439b985c56442c3ae442" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:21.118674 11531 consensus_queue.cc:1035] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3de5b4460a14551b9b216747223f98e" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:21.152128 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.146222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:34489 (local address 127.2.223.129:44971)
0807 11:17:21.146395 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:21.146401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:21.146542 (+ 141us) server_negotiation.cc:407] Connection header received
0807 11:17:21.146850 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:21.146855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:21.146951 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:21.147030 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:21.147813 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.148764 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.150066 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.150809 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.150933 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.151315 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:21.151472 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:21.151568 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:21.151729 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 11:17:21.151811 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:17:21.152237 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.145806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44971 (local address 127.2.223.130:34489)
0807 11:17:21.146417 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:21.146437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:21.146671 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:21.147304 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:21.147309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:21.147324 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:17:21.147623 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:21.147640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.148919 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.148923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:21.149913 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:21.149924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.150963 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.150967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:21.151102 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.151139 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:17:21.151592 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:17:21.151605 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 11:17:21.151695 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 11:17:21.151844 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":2179,"threads_started":1}
I20260807 11:17:21.183137 10985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:21.183403 10985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:21.186841 10985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f27650aa94dc40ad9a4c671c8688da40: Sending DeleteTablet for 3 replicas of tablet 67bf1d112cd746c7984eff41a9132b8d
I20260807 11:17:21.187812 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d112cd746c7984eff41a9132b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:21 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:57158
I20260807 11:17:21.187868 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d112cd746c7984eff41a9132b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:21 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:47238
I20260807 11:17:21.188126 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d112cd746c7984eff41a9132b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:21 UTC) from {username='oryx', principal='oryx/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60114
I20260807 11:17:21.188215 11550 tablet_replica.cc:330] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: stopping tablet replica
I20260807 11:17:21.188444 11552 tablet_replica.cc:330] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: stopping tablet replica
I20260807 11:17:21.188421 11550 raft_consensus.cc:2229] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:21.188680 11552 raft_consensus.cc:2229] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:21.188783 11550 raft_consensus.cc:2258] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:21.188890 11552 raft_consensus.cc:2258] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:21.189105 11551 tablet_replica.cc:330] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: stopping tablet replica
I20260807 11:17:21.189318 11551 raft_consensus.cc:2229] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:21.189513 11551 raft_consensus.cc:2258] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:21.190869 11550 ts_tablet_manager.cc:1905] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:21.190876 11552 ts_tablet_manager.cc:1905] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:21.191062 11551 ts_tablet_manager.cc:1905] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:21.195389 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11063
I20260807 11:17:21.198467 11551 ts_tablet_manager.cc:1918] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:21.198652 11551 log.cc:1196] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/67bf1d112cd746c7984eff41a9132b8d
I20260807 11:17:21.199105 11551 ts_tablet_manager.cc:1939] T 67bf1d112cd746c7984eff41a9132b8d P 7bd1ce5ccc10439b985c56442c3ae442: Deleting consensus metadata
I20260807 11:17:21.199720 11550 ts_tablet_manager.cc:1918] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:21.199891 11550 log.cc:1196] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/67bf1d112cd746c7984eff41a9132b8d
I20260807 11:17:21.200258 10972 catalog_manager.cc:4841] TS 7bd1ce5ccc10439b985c56442c3ae442 (127.2.223.130:35073): tablet 67bf1d112cd746c7984eff41a9132b8d (table test-table [id=5f290685230a428aba0d476d8fea163e]) successfully deleted
I20260807 11:17:21.200332 11550 ts_tablet_manager.cc:1939] T 67bf1d112cd746c7984eff41a9132b8d P 4d10d0a809af40e4bc146614d8b208a8: Deleting consensus metadata
I20260807 11:17:21.201318 10975 catalog_manager.cc:4841] TS 4d10d0a809af40e4bc146614d8b208a8 (127.2.223.129:44971): tablet 67bf1d112cd746c7984eff41a9132b8d (table test-table [id=5f290685230a428aba0d476d8fea163e]) successfully deleted
I20260807 11:17:21.202863 11552 ts_tablet_manager.cc:1918] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:17:21.203051 11552 log.cc:1196] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/67bf1d112cd746c7984eff41a9132b8d
I20260807 11:17:21.203506 11552 ts_tablet_manager.cc:1939] T 67bf1d112cd746c7984eff41a9132b8d P a3de5b4460a14551b9b216747223f98e: Deleting consensus metadata
I20260807 11:17:21.204492 10972 catalog_manager.cc:4841] TS a3de5b4460a14551b9b216747223f98e (127.2.223.131:41139): tablet 67bf1d112cd746c7984eff41a9132b8d (table test-table [id=5f290685230a428aba0d476d8fea163e]) successfully deleted
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.129@KRBTEST.COM
I20260807 11:17:21.219628 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.196952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.2.223.129:44971)
0807 11:17:21.197150 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:21.197156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:21.203046 (+ 5890us) server_negotiation.cc:407] Connection header received
0807 11:17:21.203408 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:21.203414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:21.203502 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:21.203618 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:21.205400 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.206130 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.207906 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.208375 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.208472 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.213085 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:21.213102 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:21.213123 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:21.213159 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:21.216448 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.217213 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.217221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.217223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.217370 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.217807 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.217811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.217813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.218199 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:21.218658 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:21.218972 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 11:17:21.219219 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:17:21.220463 11199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:21.347750 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11063
I20260807 11:17:21.374898 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11209
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.130@KRBTEST.COM
I20260807 11:17:21.389631 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.375828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.2.223.130:35073)
0807 11:17:21.376022 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:21.376027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:21.376046 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:17:21.376105 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:21.376109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:21.376172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:21.376262 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:21.377554 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.378343 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.379594 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.380006 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.380106 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.384180 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:21.384196 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:21.384213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:21.384261 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:21.387133 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.387846 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.387854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.387856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.387983 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.388325 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.388329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.388331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.388713 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:21.388958 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:21.389187 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 11:17:21.389358 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:17:21.390362 11345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:21.526434 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11209
I20260807 11:17:21.556283 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11355
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.223.131@KRBTEST.COM
I20260807 11:17:21.572414 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:21.557001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.2.223.131:41139)
0807 11:17:21.557513 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:21.557518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:21.557532 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:17:21.557585 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:21.557589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:21.557669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:21.557844 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:21.559323 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.560052 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.561320 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:21.561782 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:21.561964 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:21.566377 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:21.566393 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:21.566411 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:21.566444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:21.569509 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.570179 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.570212 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.570217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.570378 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:21.570841 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:21.570846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:21.570850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:21.571214 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:21.571556 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:21.571858 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:17:21.572102 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":262,"threads_started":1}
I20260807 11:17:21.573024 11493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:21.703362 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11355
I20260807 11:17:21.731443 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 10956
I20260807 11:17:21.732777 11031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:21.854754 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 10956
2026-08-07T11:17:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7060 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[11635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[11635](info): set up 2 sockets
Aug 07 11:17:21 dist-test-slave-5l3k krb5kdc[11635](info): commencing operation
krb5kdc: starting...
W20260807 11:17:23.025475 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.007s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:23 dist-test-slave-5l3k krb5kdc[11635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:17:23.323280 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45135
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:23.668555 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:23.668967 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:23.669147 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:23.676796 11648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:23.676975 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:23.677091 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:23.677194 11648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:23.677292 11648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:23.687783 11648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:23.689239 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:23.690719 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:23.703207 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:23.703621 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:23.703645 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:23.704149 11648 server_base.cc:1034] running on GCE node
I20260807 11:17:23.705045 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:23.706609 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:23.707876 11648 hybrid_clock.cc:648] HybridClock initialized: now 1786101443707810 us; error 50 us; skew 500 ppm
I20260807 11:17:23.708283 11648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T11:17:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1919 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:23 dist-test-slave-5l3k krb5kdc[11662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:23 dist-test-slave-5l3k krb5kdc[11662](info): set up 2 sockets
Aug 07 11:17:23 dist-test-slave-5l3k krb5kdc[11662](info): commencing operation
krb5kdc: starting...
W20260807 11:17:24.976994 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.007s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:25 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:17:25.266005 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41473
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:25.605255 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:25.605568 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:25.605792 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:25.613047 11675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:25.613149 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:25.613274 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:25.613394 11675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:25.613474 11675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:25.623416 11675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:25.624759 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:25.626129 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:25.638182 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:25.638236 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:25.638281 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:25.639161 11675 server_base.cc:1034] running on GCE node
I20260807 11:17:25.639894 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:25.641194 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:25.642419 11675 hybrid_clock.cc:648] HybridClock initialized: now 1786101445642393 us; error 49 us; skew 500 ppm
I20260807 11:17:25.642813 11675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:25 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:25.653018 11675 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:17:25.654714 11675 webserver.cc:460] Webserver started at http://127.2.223.190:37895/ using document root <none> and password file <none>
I20260807 11:17:25.655457 11675 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:25.655622 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:25.655963 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:25.659228 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "f1ad2289387f4e3d8b2a0fde26adb886"
format_stamp: "Formatted at 2026-08-07 11:17:25 on dist-test-slave-5l3k"
server_key: "100fe0d70933eb7eaa7c76801f66828b"
server_key_iv: "c1b2bc28f74beb229b8c536084a36edb"
server_key_version: "encryptionkey@0"
I20260807 11:17:25.660071 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "f1ad2289387f4e3d8b2a0fde26adb886"
format_stamp: "Formatted at 2026-08-07 11:17:25 on dist-test-slave-5l3k"
server_key: "100fe0d70933eb7eaa7c76801f66828b"
server_key_iv: "c1b2bc28f74beb229b8c536084a36edb"
server_key_version: "encryptionkey@0"
I20260807 11:17:25.665735 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 11:17:25.669574 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:25.671140 11675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:25.671356 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "f1ad2289387f4e3d8b2a0fde26adb886"
format_stamp: "Formatted at 2026-08-07 11:17:25 on dist-test-slave-5l3k"
server_key: "100fe0d70933eb7eaa7c76801f66828b"
server_key_iv: "c1b2bc28f74beb229b8c536084a36edb"
server_key_version: "encryptionkey@0"
I20260807 11:17:25.671576 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:25.693677 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:25.698027 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:25.698354 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:25.718397 11675 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41473
I20260807 11:17:25.718434 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41473 every 8 connection(s)
I20260807 11:17:25.719857 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:25.723722 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:25.724681 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11675
I20260807 11:17:25.724907 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:25.725767 2942 external_mini_cluster.cc:1428] Setting key 3a25cafd2319c15480565caa354ca8a1
I20260807 11:17:25.736876 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Bootstrap starting.
Aug 07 11:17:25 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:25.740346 11746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:25.741438 11746 log.cc:824] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:25.744715 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: No bootstrap required, opened a new log
I20260807 11:17:25.748788 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:25.728110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.2.223.190:41473)
0807 11:17:25.728896 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:25.728908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:25.728950 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:17:25.730667 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:25.730689 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:25.731086 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:25.731488 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:25.732992 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:25.734319 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:25.735653 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:25.736314 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:25.736501 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:25.740958 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:25.740967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:25.740983 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:25.741022 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:25.744423 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:25.745250 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:25.745257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:25.745264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:25.745443 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:25.745870 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:25.745875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:25.745878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:25.746411 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:25.746838 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:25.747277 (+ 439us) server_negotiation.cc:299] Negotiation successful
0807 11:17:25.747744 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260807 11:17:25.753103 11746 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:25.753615 11746 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:25.753736 11746 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad2289387f4e3d8b2a0fde26adb886, State: Initialized, Role: FOLLOWER
I20260807 11:17:25.754386 11746 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:25.754700 11746 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:25.754853 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:25.755028 11746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:25.758687 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:25.759322 11746 leader_election.cc:302] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ad2289387f4e3d8b2a0fde26adb886; no voters:
I20260807 11:17:25.760110 11746 leader_election.cc:288] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:25.760385 11751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:25.761310 11751 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 LEADER]: Becoming Leader. State: Replica: f1ad2289387f4e3d8b2a0fde26adb886, State: Running, Role: LEADER
I20260807 11:17:25.762073 11751 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:25.764253 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ad2289387f4e3d8b2a0fde26adb886. Latest consensus state: current_term: 1 leader_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } } }
I20260807 11:17:25.764868 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:25.767612 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } } }
I20260807 11:17:25.767973 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:25.768632 11746 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:25.774340 11755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:25.782554 11755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:25.791263 11755 catalog_manager.cc:1347] Generated new cluster ID: 34862d54c4ef4ab4bf5a00444ead8e0c
I20260807 11:17:25.791430 11755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:25.811404 11755 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:25.814006 11755 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:25.834054 11755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Generated new TSK 0
I20260807 11:17:25.835335 11755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:17:25.918555 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:26.264709 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:26.264994 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:26.265110 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:26.272902 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:26.273033 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:26.273150 11773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:26.283293 11773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:26.284591 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:26.286005 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:26.300076 11781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:26.299996 11778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:26.300040 11779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:26.300921 11773 server_base.cc:1034] running on GCE node
I20260807 11:17:26.301542 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:26.302600 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:26.303838 11773 hybrid_clock.cc:648] HybridClock initialized: now 1786101446303768 us; error 61 us; skew 500 ppm
I20260807 11:17:26.304215 11773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:26 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:26.307813 11773 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:17:26.309507 11773 webserver.cc:460] Webserver started at http://127.2.223.129:46239/ using document root <none> and password file <none>
I20260807 11:17:26.310277 11773 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:26.310433 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:26.310820 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:26.313889 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "ff0a2624cdc444e78606b7b472893ab0"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "be267458e4e5bee494f52e0d236d1332"
server_key_iv: "291e60ef5d2dc972548dd3853dd5e68c"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.314785 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "ff0a2624cdc444e78606b7b472893ab0"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "be267458e4e5bee494f52e0d236d1332"
server_key_iv: "291e60ef5d2dc972548dd3853dd5e68c"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.320333 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:26.323931 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:26.325280 11773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:26.325456 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "ff0a2624cdc444e78606b7b472893ab0"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "be267458e4e5bee494f52e0d236d1332"
server_key_iv: "291e60ef5d2dc972548dd3853dd5e68c"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.325675 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:26.343290 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:26.347538 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:26.347846 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:26.348827 11773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:26.350266 11773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:26.350366 11773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:26.350555 11773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:26.350633 11773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 11:17:26.386217 11773 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:33491
I20260807 11:17:26.386270 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:33491 every 8 connection(s)
I20260807 11:17:26.387624 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:26.398110 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11773
I20260807 11:17:26.398384 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:26.399326 2942 external_mini_cluster.cc:1428] Setting key 940c5e72cecf94cebedf042709473918
Aug 07 11:17:26 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:26.408947 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:26.390462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:56361 (local address 127.2.223.190:41473)
0807 11:17:26.391029 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:26.391034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:26.391773 (+ 739us) server_negotiation.cc:407] Connection header received
0807 11:17:26.393636 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:26.393641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:26.393735 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:26.393859 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:26.395475 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.396220 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.397816 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.398217 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.398376 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:26.402842 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:26.402847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:26.402851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:26.402883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:26.405623 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:26.406418 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:26.406423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:26.406425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:26.406614 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:26.407095 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:26.407100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:26.407104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:26.407381 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:26.407571 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:26.408326 (+ 755us) server_negotiation.cc:299] Negotiation successful
0807 11:17:26.408571 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260807 11:17:26.410181 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:26.390897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:56361)
0807 11:17:26.391614 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:26.391654 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:26.393367 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:26.394116 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:26.394125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:26.394727 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:26.395297 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:26.395316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.396401 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.396404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:26.397662 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:26.397673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.398523 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.398534 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:26.398872 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:26.399706 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:26.399728 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:26.402605 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:26.405776 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:26.405786 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:26.405803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:26.406273 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:26.406734 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:26.406739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:26.406743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:26.406959 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:26.407522 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:26.407538 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:26.407913 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 11:17:26.408201 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 11:17:26.408799 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":261,"threads_started":1}
I20260807 11:17:26.412009 11904 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41473
I20260807 11:17:26.412367 11904 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:26.413188 11904 heartbeater.cc:511] Master 127.2.223.190:41473 requested a full tablet report, sending...
I20260807 11:17:26.415691 11708 ts_manager.cc:194] Registered new tserver with Master: ff0a2624cdc444e78606b7b472893ab0 (127.2.223.129:33491)
I20260807 11:17:26.417258 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:56361
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:17:26.463537 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:26.817054 11910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:26.817363 11910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:26.817551 11910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:26.825253 11910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:26.825412 11910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:26.825623 11910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:26.835994 11910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:26.837384 11910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:26.838841 11910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:26.854387 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:26.855033 11916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:26.855232 11918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:26.855912 11910 server_base.cc:1034] running on GCE node
I20260807 11:17:26.856635 11910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:26.857661 11910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:26.858920 11910 hybrid_clock.cc:648] HybridClock initialized: now 1786101446858883 us; error 53 us; skew 500 ppm
I20260807 11:17:26.859272 11910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:26 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:26.862638 11910 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:17:26.864228 11910 webserver.cc:460] Webserver started at http://127.2.223.130:43465/ using document root <none> and password file <none>
I20260807 11:17:26.864949 11910 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:26.865097 11910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:26.865438 11910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:26.868572 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "bb4a79186452422da9d79f8e4cb8a72d"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "dcaff9639d3d306db4ea3d21ee456b4b"
server_key_iv: "d9e27df2ea3955358ee90091324ddef6"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.869453 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "bb4a79186452422da9d79f8e4cb8a72d"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "dcaff9639d3d306db4ea3d21ee456b4b"
server_key_iv: "d9e27df2ea3955358ee90091324ddef6"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.875226 11910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:26.879218 11925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:26.880731 11910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:26.880941 11910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "bb4a79186452422da9d79f8e4cb8a72d"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "dcaff9639d3d306db4ea3d21ee456b4b"
server_key_iv: "d9e27df2ea3955358ee90091324ddef6"
server_key_version: "encryptionkey@0"
I20260807 11:17:26.881186 11910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:26.907928 11910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:26.912634 11910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:26.912951 11910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:26.914036 11910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:26.915470 11910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:26.915584 11910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:26.915745 11910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:26.915820 11910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:26.951303 11910 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:33601
I20260807 11:17:26.951393 12041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:33601 every 8 connection(s)
I20260807 11:17:26.952646 11910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:26.954660 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 11910
I20260807 11:17:26.954850 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:26.955322 2942 external_mini_cluster.cc:1428] Setting key f685d349b7171a479ec0170bc46f4161
Aug 07 11:17:26 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101446, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:26.979573 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:26.955591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:40095 (local address 127.2.223.190:41473)
0807 11:17:26.956158 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:26.956166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:26.957357 (+ 1191us) server_negotiation.cc:407] Connection header received
0807 11:17:26.960750 (+ 3393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:26.960760 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:26.960880 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:26.961015 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:26.963529 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.964772 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.967475 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.967933 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.968042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:26.973024 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:26.973029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:26.973033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:26.973070 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:26.976174 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:26.976991 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:26.976997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:26.977000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:26.977157 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:26.977630 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:26.977635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:26.977638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:26.977937 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:26.978100 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:26.978925 (+ 825us) server_negotiation.cc:299] Negotiation successful
0807 11:17:26.979173 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":271,"threads_started":1}
I20260807 11:17:26.980746 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:26.956271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:40095)
0807 11:17:26.957178 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:26.957225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:26.960358 (+ 3133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:26.961392 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:26.961408 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:26.962344 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:26.963278 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:26.963308 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.964988 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.964997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:26.967306 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:26.967327 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:26.968199 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:26.968210 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:26.968592 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:26.969519 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:26.969557 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:26.972733 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:26.976355 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:26.976363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:26.976378 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:26.976845 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:26.977278 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:26.977286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:26.977294 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:26.977501 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:26.978088 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:26.978096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:26.978459 (+ 363us) client_negotiation.cc:769] Sending connection context
0807 11:17:26.978826 (+ 367us) client_negotiation.cc:240] Negotiation successful
0807 11:17:26.979421 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":381,"threads_started":1}
I20260807 11:17:26.982678 12042 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41473
I20260807 11:17:26.983016 12042 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:26.983806 12042 heartbeater.cc:511] Master 127.2.223.190:41473 requested a full tablet report, sending...
I20260807 11:17:26.985656 11708 ts_manager.cc:194] Registered new tserver with Master: bb4a79186452422da9d79f8e4cb8a72d (127.2.223.130:33601)
I20260807 11:17:26.987073 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:40095
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:17:27.025763 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:27.373756 12048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:27.374030 12048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:27.374184 12048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:27.381814 12048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:27.381928 12048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:27.382110 12048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:27.393122 12048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:27.394531 12048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:27.395859 12048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:27.409107 12053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:27.409381 12054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:27.410300 12057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:27.410697 12048 server_base.cc:1034] running on GCE node
I20260807 11:17:27.411345 12048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:27.412439 12048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:27.413672 12048 hybrid_clock.cc:648] HybridClock initialized: now 1786101447413623 us; error 42 us; skew 500 ppm
I20260807 11:17:27.414099 12048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:27 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:27.417425 12048 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:17:27.419230 12048 webserver.cc:460] Webserver started at http://127.2.223.131:45859/ using document root <none> and password file <none>
I20260807 11:17:27.419968 12048 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:27.420110 12048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:27.420564 12048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:27.420642 11904 heartbeater.cc:503] Master 127.2.223.190:41473 was elected leader, sending a full tablet report...
I20260807 11:17:27.424381 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "aaec00afc30f4eaca4a8d0611cf6438a"
format_stamp: "Formatted at 2026-08-07 11:17:27 on dist-test-slave-5l3k"
server_key: "0ee3b777e8123932440e286672c61500"
server_key_iv: "c03ee734f52e4ce255163ffcd239062b"
server_key_version: "encryptionkey@0"
I20260807 11:17:27.425220 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "aaec00afc30f4eaca4a8d0611cf6438a"
format_stamp: "Formatted at 2026-08-07 11:17:27 on dist-test-slave-5l3k"
server_key: "0ee3b777e8123932440e286672c61500"
server_key_iv: "c03ee734f52e4ce255163ffcd239062b"
server_key_version: "encryptionkey@0"
I20260807 11:17:27.430936 12048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:27.434746 12064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:27.436200 12048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:27.436360 12048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "aaec00afc30f4eaca4a8d0611cf6438a"
format_stamp: "Formatted at 2026-08-07 11:17:27 on dist-test-slave-5l3k"
server_key: "0ee3b777e8123932440e286672c61500"
server_key_iv: "c03ee734f52e4ce255163ffcd239062b"
server_key_version: "encryptionkey@0"
I20260807 11:17:27.436550 12048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:27.451519 12048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:27.456270 12048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:27.456588 12048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:27.457701 12048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:27.459223 12048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:27.459297 12048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:27.459460 12048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:27.459512 12048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:27.506414 12048 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37603
I20260807 11:17:27.506453 12180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37603 every 8 connection(s)
I20260807 11:17:27.507825 12048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:27.516466 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12048
I20260807 11:17:27.516702 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:27.517220 2942 external_mini_cluster.cc:1428] Setting key 24c99d5dc23813186e24024c58ec3f2a
Aug 07 11:17:27 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:27.535808 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:27.510775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:59047 (local address 127.2.223.190:41473)
0807 11:17:27.511413 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:27.511423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:27.512595 (+ 1172us) server_negotiation.cc:407] Connection header received
0807 11:17:27.515988 (+ 3393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:27.515996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:27.516127 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:27.516284 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:27.518684 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.519755 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.522691 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.523208 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.523320 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:27.528169 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:27.528174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:27.528178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:27.528216 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:27.531534 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.532892 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.532899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.532905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.533084 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.533691 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.533694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.533696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.534081 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:27.534220 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:27.535338 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0807 11:17:27.535518 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260807 11:17:27.537614 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:27.511412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:59047)
0807 11:17:27.512362 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:27.512419 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:27.515550 (+ 3131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:27.516585 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:27.516600 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:27.517471 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:27.518413 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:27.518444 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.519939 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.519948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:27.522524 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:27.522545 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.523542 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.523552 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:27.523949 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:27.524803 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:27.524843 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:27.527891 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:27.531742 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:27.531755 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:27.531780 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:27.532560 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:27.533252 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:27.533257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:27.533259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:27.533540 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:27.534233 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:27.534245 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:27.534838 (+ 593us) client_negotiation.cc:769] Sending connection context
0807 11:17:27.535216 (+ 378us) client_negotiation.cc:240] Negotiation successful
0807 11:17:27.535952 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":339,"threads_started":1}
I20260807 11:17:27.539284 12181 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41473
I20260807 11:17:27.539630 12181 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:27.540480 12181 heartbeater.cc:511] Master 127.2.223.190:41473 requested a full tablet report, sending...
I20260807 11:17:27.542189 11708 ts_manager.cc:194] Registered new tserver with Master: aaec00afc30f4eaca4a8d0611cf6438a (127.2.223.131:37603)
I20260807 11:17:27.543635 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:59047
I20260807 11:17:27.546702 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
Entry for principal oryx/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab.
WARNING: no policy specified for oryx/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
Entry for principal oryx/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab.
WARNING: no policy specified for oryx/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
Entry for principal oryx/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab.
I20260807 11:17:27.738849 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11773
Aug 07 11:17:27 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:17:27.756628 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:27.740245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.2.223.129:33491)
0807 11:17:27.740926 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:27.740933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:27.740967 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:17:27.741037 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:27.741044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:27.741338 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:27.741483 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:27.743018 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.744509 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.745838 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.746456 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.746655 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:27.750867 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:27.750881 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:27.750898 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:27.750935 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:27.753875 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.754620 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.754629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.754631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.754777 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.755135 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.755139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.755141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.755553 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:27.755812 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:27.756077 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 11:17:27.756295 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":355,"threads_started":1}
I20260807 11:17:27.758010 11899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:27.846771 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11773
I20260807 11:17:27.867715 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11910
Aug 07 11:17:27 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:17:27.884943 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:27.868678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.2.223.130:33601)
0807 11:17:27.869315 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:27.869322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:27.869348 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:17:27.869411 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:27.869417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:27.869693 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:27.869825 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:27.871268 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.872382 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.873606 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:27.874145 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:27.874243 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:27.878849 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:27.878863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:27.878880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:27.878929 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:27.881928 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.882733 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.882747 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.882753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.882934 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:27.883328 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:27.883332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:27.883334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:27.883711 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:27.883954 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:27.884406 (+ 452us) server_negotiation.cc:299] Negotiation successful
0807 11:17:27.884626 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":345,"threads_started":1}
I20260807 11:17:27.886288 12036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:27.976754 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11910
I20260807 11:17:27.997082 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12048
Aug 07 11:17:28 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:17:28.014282 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:27.998234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.2.223.131:37603)
0807 11:17:27.998804 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:27.998812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:27.998843 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:17:27.998906 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:27.998913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:27.999143 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:27.999322 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:28.000712 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:28.001683 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:28.003142 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:28.003705 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:28.003862 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:28.008399 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:28.008411 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:28.008427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:28.008462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:28.011349 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:28.012048 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:28.012056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:28.012058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:28.012194 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:28.012587 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:28.012593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:28.012596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:28.012961 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:28.013303 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:28.013621 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 11:17:28.013899 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":290,"threads_started":1}
I20260807 11:17:28.015681 12176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:28.101541 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12048
I20260807 11:17:28.121959 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 11675
I20260807 11:17:28.123498 11741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:28.232151 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 11675
I20260807 11:17:28.258569 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41473
--webserver_interface=127.2.223.190
--webserver_port=37895
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:28.613360 12200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:28.613761 12200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:28.613945 12200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:28.621515 12200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:28.621727 12200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:28.621843 12200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:28.621953 12200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:28.622041 12200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:28.632905 12200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=37895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:28.634392 12200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:28.635797 12200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:28.648411 12205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:28.648378 12206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:28.648334 12208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:28.649657 12200 server_base.cc:1034] running on GCE node
I20260807 11:17:28.650592 12200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:28.652117 12200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:28.653366 12200 hybrid_clock.cc:648] HybridClock initialized: now 1786101448653308 us; error 47 us; skew 500 ppm
I20260807 11:17:28.653748 12200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:28 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101448, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:28.657119 12200 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:17:28.658895 12200 webserver.cc:460] Webserver started at http://127.2.223.190:37895/ using document root <none> and password file <none>
I20260807 11:17:28.659653 12200 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:28.659826 12200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:28.665578 12200 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 11:17:28.668573 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:28.670083 12200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:28.670249 12200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "f1ad2289387f4e3d8b2a0fde26adb886"
format_stamp: "Formatted at 2026-08-07 11:17:25 on dist-test-slave-5l3k"
server_key: "100fe0d70933eb7eaa7c76801f66828b"
server_key_iv: "c1b2bc28f74beb229b8c536084a36edb"
server_key_version: "encryptionkey@0"
I20260807 11:17:28.671411 12200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:28.695909 12200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:28.700524 12200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:28.700765 12200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:28.720975 12200 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41473
I20260807 11:17:28.721009 12268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41473 every 8 connection(s)
I20260807 11:17:28.722661 12200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:28.729887 12269 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:17:28.730386 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12200
I20260807 11:17:28.731222 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:33491
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:41473
--webserver_port=46239
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:17:28.733397 12269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Bootstrap starting.
I20260807 11:17:28.753394 12269 log.cc:824] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:28.762887 12269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:17:28.763461 12269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Bootstrap complete.
I20260807 11:17:28.772850 12269 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:28.773754 12269 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad2289387f4e3d8b2a0fde26adb886, State: Initialized, Role: FOLLOWER
I20260807 11:17:28.774453 12269 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:28.774784 12269 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:28.774932 12269 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:28.775147 12269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:17:28.779373 12269 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:28.779894 12269 leader_election.cc:302] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ad2289387f4e3d8b2a0fde26adb886; no voters:
I20260807 11:17:28.780550 12269 leader_election.cc:288] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:17:28.780721 12273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:17:28.781850 12273 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [term 2 LEADER]: Becoming Leader. State: Replica: f1ad2289387f4e3d8b2a0fde26adb886, State: Running, Role: LEADER
I20260807 11:17:28.782505 12273 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } }
I20260807 11:17:28.782898 12269 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:28.785013 12275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ad2289387f4e3d8b2a0fde26adb886. Latest consensus state: current_term: 2 leader_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } } }
I20260807 11:17:28.785391 12275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:28.785928 12274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad2289387f4e3d8b2a0fde26adb886" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41473 } } }
I20260807 11:17:28.786253 12274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:28.797996 12280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:28.804474 12280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:28.806021 12280 catalog_manager.cc:1259] Loaded cluster ID: 34862d54c4ef4ab4bf5a00444ead8e0c
I20260807 11:17:28.806200 12280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:28.810453 12280 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:28.811981 12280 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1ad2289387f4e3d8b2a0fde26adb886: Loaded TSK: 0
I20260807 11:17:28.813829 12280 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:17:29.098359 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:29.098732 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:29.098909 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:29.106352 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:29.106539 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:29.106722 12271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:29.116683 12271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.129.keytab
--principal=oryx/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=46239
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:29.118127 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:29.119614 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:29.135025 12296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:29.135106 12298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:29.134968 12295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:29.136963 12271 server_base.cc:1034] running on GCE node
I20260807 11:17:29.137603 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:29.138703 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:29.139937 12271 hybrid_clock.cc:648] HybridClock initialized: now 1786101449139888 us; error 46 us; skew 500 ppm
I20260807 11:17:29.140266 12271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101449, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:29.143851 12271 init.cc:377] Logged in from keytab as oryx/127.2.223.129@KRBTEST.COM (short username oryx)
I20260807 11:17:29.145404 12271 webserver.cc:460] Webserver started at http://127.2.223.129:46239/ using document root <none> and password file <none>
I20260807 11:17:29.146117 12271 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:29.146289 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:29.151824 12271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:17:29.154683 12305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.156194 12271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:17:29.156373 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "ff0a2624cdc444e78606b7b472893ab0"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "be267458e4e5bee494f52e0d236d1332"
server_key_iv: "291e60ef5d2dc972548dd3853dd5e68c"
server_key_version: "encryptionkey@0"
I20260807 11:17:29.157375 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:29.184058 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:29.188386 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:29.188675 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:29.189754 12271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:29.191248 12271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:29.191370 12271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.191512 12271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:29.191605 12271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.227258 12271 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:33491
I20260807 11:17:29.227350 12421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:33491 every 8 connection(s)
I20260807 11:17:29.228791 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:29.232103 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12271
I20260807 11:17:29.232883 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:33601
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:41473
--webserver_port=43465
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:17:29.611663 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:29.611990 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:29.612182 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:29.619361 12426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:29.619509 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:29.619686 12426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:29.629429 12426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.130.keytab
--principal=oryx/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:33601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=43465
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:29.630885 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:29.632264 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:29.648499 12437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:29.648559 12435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:29.650455 12426 server_base.cc:1034] running on GCE node
W20260807 11:17:29.650496 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:29.651358 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:29.652438 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:29.653766 12426 hybrid_clock.cc:648] HybridClock initialized: now 1786101449653692 us; error 76 us; skew 500 ppm
I20260807 11:17:29.654125 12426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101449, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:29.657842 12426 init.cc:377] Logged in from keytab as oryx/127.2.223.130@KRBTEST.COM (short username oryx)
I20260807 11:17:29.659660 12426 webserver.cc:460] Webserver started at http://127.2.223.130:43465/ using document root <none> and password file <none>
I20260807 11:17:29.660410 12426 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:29.660640 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:29.666460 12426 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:17:29.669399 12444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.671037 12426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:17:29.671223 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "bb4a79186452422da9d79f8e4cb8a72d"
format_stamp: "Formatted at 2026-08-07 11:17:26 on dist-test-slave-5l3k"
server_key: "dcaff9639d3d306db4ea3d21ee456b4b"
server_key_iv: "d9e27df2ea3955358ee90091324ddef6"
server_key_version: "encryptionkey@0"
I20260807 11:17:29.672238 12426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:29.690204 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:29.694985 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:29.695292 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:29.696421 12426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:29.697950 12426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:29.698102 12426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.698239 12426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:29.698315 12426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:29.745515 12426 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:33601
I20260807 11:17:29.745579 12559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:33601 every 8 connection(s)
I20260807 11:17:29.747159 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:29.754931 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12426
I20260807 11:17:29.756063 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:37603
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:41473
--webserver_port=45859
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:29 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:17:30.121168 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:30.121475 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:30.121656 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:30.129071 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:30.129212 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:30.129429 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:30.139140 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/krb5kdc/oryx_127.2.223.131.keytab
--principal=oryx/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:37603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=45859
--tserver_master_addrs=127.2.223.190:41473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:30.140450 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:30.141793 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:30.175588 12578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:30.176594 12579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:30.192759 12581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:30.201215 12572 server_base.cc:1034] running on GCE node
I20260807 11:17:30.202600 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:30.209452 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:30.211047 12572 hybrid_clock.cc:648] HybridClock initialized: now 1786101450210935 us; error 92 us; skew 500 ppm
I20260807 11:17:30.211706 12572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101450, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:30.218286 12572 init.cc:377] Logged in from keytab as oryx/127.2.223.131@KRBTEST.COM (short username oryx)
I20260807 11:17:30.244643 12572 webserver.cc:460] Webserver started at http://127.2.223.131:45859/ using document root <none> and password file <none>
I20260807 11:17:30.246749 12572 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:30.247103 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:30.260536 12572 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260807 11:17:30.267204 12588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:30.270002 12572 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260807 11:17:30.270414 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "aaec00afc30f4eaca4a8d0611cf6438a"
format_stamp: "Formatted at 2026-08-07 11:17:27 on dist-test-slave-5l3k"
server_key: "0ee3b777e8123932440e286672c61500"
server_key_iv: "c03ee734f52e4ce255163ffcd239062b"
server_key_version: "encryptionkey@0"
I20260807 11:17:30.273154 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:30.314175 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:30.322889 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:30.323338 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:30.325160 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:30.327991 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:30.328161 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:30.328320 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:30.328442 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:30.391813 12572 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:37603
I20260807 11:17:30.391932 12708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:37603 every 8 connection(s)
I20260807 11:17:30.394655 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:30.395592 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12572
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:30.470433 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:30.397018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.2.223.190:41473)
0807 11:17:30.419299 (+ 22281us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x60c00006c200. stack: 00007f0cbde6366e 00007f0cbde63430 00007f0cbceea5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f0cd0d46775 00007f0cc8d5d270 00007f0cbde87358 00000000009da5bb 00007f0cbdea6fd7 00007f0cbdea2bea 00007f0cbdea7cee 00007f0cc4916e4a 00007f0cc4916377 00007f0cc492732d
0807 11:17:30.419566 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:30.419575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:30.438877 (+ 19302us) server_negotiation.cc:407] Connection header received
0807 11:17:30.439135 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:30.439143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:30.439252 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:30.439374 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:30.442957 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.445228 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.456528 (+ 11300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.457660 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.457840 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.461387 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:30.461394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:30.461410 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:30.461456 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:30.465744 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:30.466448 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:30.466457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:30.466468 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:30.466722 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:30.467134 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:30.467140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:30.467145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:30.467819 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:30.468163 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:30.468602 (+ 439us) server_negotiation.cc:299] Negotiation successful
0807 11:17:30.469259 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":546,"server-negotiator.queue_time_us":13434,"spinlock_wait_cycles":18326272,"thread_start_us":13506,"threads_started":1}
W20260807 11:17:30.829998 12425 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.584s user 0.005s sys 0.011s
W20260807 11:17:30.832345 12424 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:29.232198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:34551 (local address 127.2.223.190:41473)
0807 11:17:29.232989 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:29.233006 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:29.233957 (+ 951us) server_negotiation.cc:407] Connection header received
0807 11:17:29.237260 (+ 3303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:29.237273 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:29.237622 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:29.237932 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:29.240059 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.241331 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.243866 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.244749 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.244947 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.831827 (+1586880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:34551: BlockingRecv error: recv got EOF from 127.2.223.129:34551 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":283,"threads_started":1}
I20260807 11:17:30.833688 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:29.232230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:34551)
0807 11:17:29.233733 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:29.233799 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:29.236644 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:29.238161 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:29.238175 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:29.239024 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:29.239846 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:29.239873 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.241515 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.241523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:29.243688 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:29.243707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.244974 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.244979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:29.245177 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:29.245891 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:29.245914 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:30.831396 (+1585482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":994,"threads_started":1}
W20260807 11:17:30.835189 12422 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:41473 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.223.190:41473: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:30.864573 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:30.836257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:57893)
0807 11:17:30.836645 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:30.836674 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:30.836834 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:30.837374 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:30.837382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:30.838158 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:30.838801 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.838826 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.841709 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.841718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.844117 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.844139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.845572 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.845582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.845856 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.858588 (+ 12732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:30.858621 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:30.864247 (+ 5626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210}
W20260807 11:17:30.864763 12424 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:30.836500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:57893 (local address 127.2.223.190:41473)
0807 11:17:30.836744 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:30.836754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:30.836783 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:17:30.837015 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:30.837024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:30.837152 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:30.837311 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:30.839315 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.841138 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.844327 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.845313 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.845504 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.864494 (+ 18990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:57893: BlockingRecv error: recv got EOF from 127.2.223.129:57893 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:30 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:30.875149 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:30.865553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:34165)
0807 11:17:30.865949 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:30.865968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:30.866092 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:30.866505 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:30.866510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:30.866866 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:30.867026 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.867034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.868105 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.868111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.869149 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.869164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.870340 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.870346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.870573 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.871260 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:30.871284 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:30.874747 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285}
W20260807 11:17:30.875298 12424 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:30.865743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:34165 (local address 127.2.223.190:41473)
0807 11:17:30.865928 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:30.865938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:30.866018 (+ 80us) server_negotiation.cc:407] Connection header received
0807 11:17:30.866262 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:30.866270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:30.866338 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:30.866442 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:30.867185 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.867956 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.869346 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.870007 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.870123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.874963 (+ 4840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:34165: BlockingRecv error: recv got EOF from 127.2.223.129:34165 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 11:17:30.875870 12422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:17:31.230401 12571 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.468s user 0.008s sys 0.010s
W20260807 11:17:31.231766 12564 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:29.750318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48787 (local address 127.2.223.190:41473)
0807 11:17:29.750844 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:29.750849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:29.752828 (+ 1979us) server_negotiation.cc:407] Connection header received
0807 11:17:29.755124 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:29.755130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:29.755265 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:29.755415 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:29.757463 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.758319 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.760098 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.760545 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.760711 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.231453 (+1470742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:48787: BlockingRecv error: recv got EOF from 127.2.223.130:48787 (error 108)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
I20260807 11:17:31.232441 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:29.751482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:48787)
0807 11:17:29.752596 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:29.752650 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:29.754771 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:29.755769 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:29.755780 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:29.756546 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:29.757218 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:29.757242 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.758567 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.758572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:29.759905 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:29.759919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:29.760937 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:29.760954 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:29.761205 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:29.762073 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:29.762100 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.231259 (+1469159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":478,"threads_started":1}
W20260807 11:17:31.233691 12561 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:41473 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.223.190:41473: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:31.246537 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:31.234601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:52089)
0807 11:17:31.234799 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:31.234821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:31.234935 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:31.235314 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:31.235318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:31.235707 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:31.235870 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.235879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.237037 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.237047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.239473 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.239496 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.240282 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.240292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.240575 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.241367 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:31.241404 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.246218 (+ 4814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 11:17:31.246670 12564 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:31.234701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:52089 (local address 127.2.223.190:41473)
0807 11:17:31.234914 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:31.234919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:31.234938 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:17:31.235104 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:31.235109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:31.235182 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:31.235282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:31.236015 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.236817 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.239653 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.240038 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.240154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.246403 (+ 6249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:52089: BlockingRecv error: recv got EOF from 127.2.223.130:52089 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:17:31.259084 12564 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:31.247543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43451 (local address 127.2.223.190:41473)
0807 11:17:31.247693 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:31.247706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:31.247721 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:17:31.247964 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:31.247975 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:31.248071 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:31.248168 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:31.249403 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.251145 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.252427 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.253382 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.253552 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.258830 (+ 5278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:43451: BlockingRecv error: recv got EOF from 127.2.223.130:43451 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 11:17:31.259073 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:31.247383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:43451)
0807 11:17:31.247638 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:31.247666 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:31.247806 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:31.248246 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:31.248253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:31.248842 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:31.249161 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.249178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.251304 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.251307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.252274 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.252285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.253600 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.253605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.253740 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.254398 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:31.254415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.258606 (+ 4191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260807 11:17:31.259454 12561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:17:31.729300 12713 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.264s user 0.007s sys 0.013s
W20260807 11:17:31.730634 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:30.424822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:48605 (local address 127.2.223.190:41473)
0807 11:17:30.445860 (+ 21038us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:30.445868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:30.447483 (+ 1615us) server_negotiation.cc:407] Connection header received
0807 11:17:30.451578 (+ 4095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:30.451588 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:30.451719 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:30.451866 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:30.455262 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.458073 (+ 2811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.460965 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.462899 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.463032 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.730305 (+1267273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:48605: BlockingRecv error: recv got EOF from 127.2.223.131:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":20842,"thread_start_us":21047,"threads_started":1}
I20260807 11:17:31.731278 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:30.423976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:48605)
0807 11:17:30.446931 (+ 22955us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:30.447002 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:30.450897 (+ 3895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:30.452619 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:30.452641 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:30.453958 (+ 1317us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:30.455048 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.455079 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.458263 (+ 3184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.458269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.460645 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:30.460665 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:30.463691 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:30.463711 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:30.464053 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:30.464817 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:30.464848 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.730131 (+1265283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22594,"thread_start_us":22301,"threads_started":1}
W20260807 11:17:31.732261 12709 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:41473 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.223.190:41473: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:31.741585 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:31.733026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:57775)
0807 11:17:31.733225 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:31.733242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:31.733340 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:31.733656 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:31.733659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:31.733993 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:31.734192 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.734201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.735390 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.735396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.736375 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.736388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.737123 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.737129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.737287 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.737835 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:31.737852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.741279 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 11:17:31.741737 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:31.733085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:57775 (local address 127.2.223.190:41473)
0807 11:17:31.733247 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:31.733251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:31.733299 (+ 48us) server_negotiation.cc:407] Connection header received
0807 11:17:31.733433 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:31.733437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:31.733498 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:31.733660 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:31.734312 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.735241 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.736522 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.736929 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.737074 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.741470 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:57775: BlockingRecv error: recv got EOF from 127.2.223.131:57775 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:31.750741 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:31.742351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:50613)
0807 11:17:31.742558 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:31.742586 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:31.742677 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:31.742935 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:31.742938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:31.743201 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:31.743340 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.743347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.744302 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.744305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.745314 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.745326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.746281 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.746286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.746467 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.747129 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:31.747153 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.750471 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 11:17:31.750973 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:31.742432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:50613 (local address 127.2.223.190:41473)
0807 11:17:31.742699 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:31.742703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:31.742718 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:17:31.742756 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:31.742759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:31.742819 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:31.742893 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:31.743451 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.744174 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.745460 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.746120 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.746253 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.750712 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:50613: BlockingRecv error: recv got EOF from 127.2.223.131:50613 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
W20260807 11:17:31.751080 12709 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:31 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:31.886341 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:31.876513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:45393)
0807 11:17:31.877289 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:31.877307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:31.877430 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:31.877836 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:31.877841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:31.878518 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:31.878755 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.878770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.879899 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.879905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.880960 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:31.880972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.881723 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.881728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:31.881846 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.882559 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:31.882599 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:31.886098 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":521,"threads_started":1}
W20260807 11:17:31.886549 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:31.876602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:45393 (local address 127.2.223.190:41473)
0807 11:17:31.876894 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:31.876901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:31.877430 (+ 529us) server_negotiation.cc:407] Connection header received
0807 11:17:31.877522 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:31.877526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:31.877606 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:31.877716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:31.878919 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.879694 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.881097 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:31.881547 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:31.881672 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:31.886307 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:45393: BlockingRecv error: recv got EOF from 127.2.223.129:45393 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:32.269569 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:32.260110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:48349)
0807 11:17:32.260702 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:32.260727 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:32.260840 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:32.261332 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:32.261336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:32.261942 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:32.262242 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.262257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.263357 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.263361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.264351 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.264363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.265050 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.265054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.265174 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.265726 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:32.265743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:32.269310 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
W20260807 11:17:32.269761 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:32.260134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48349 (local address 127.2.223.190:41473)
0807 11:17:32.260327 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:32.260331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:32.260816 (+ 485us) server_negotiation.cc:407] Connection header received
0807 11:17:32.260979 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:32.260983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:32.261081 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:32.261200 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:32.262395 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.263207 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.264472 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.264894 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.264984 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.269497 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:48349: BlockingRecv error: recv got EOF from 127.2.223.130:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:32.760895 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:32.751736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:36633)
0807 11:17:32.752269 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:32.752293 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:32.752464 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:32.752860 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:32.752865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:32.753354 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:32.753503 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.753512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.754538 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.754544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.755507 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.755519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.756316 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.756322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.756491 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.757047 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:32.757065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:32.760499 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":424,"threads_started":1}
W20260807 11:17:32.761013 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:32.751939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:36633 (local address 127.2.223.190:41473)
0807 11:17:32.752152 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:32.752158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:32.752392 (+ 234us) server_negotiation.cc:407] Connection header received
0807 11:17:32.752585 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:32.752593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:32.752719 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:32.752820 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:32.753649 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.754373 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.755667 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.756127 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.756262 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.760744 (+ 4482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:36633: BlockingRecv error: recv got EOF from 127.2.223.131:36633 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:32 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:32.896996 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:32.887548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:38635)
0807 11:17:32.888133 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:32.888149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:32.888311 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:32.888666 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:32.888670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:32.889153 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:32.889323 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.889331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.890360 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.890368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.891363 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:32.891374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.892119 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.892125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:32.892297 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.892922 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:32.892940 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:32.896737 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":314,"threads_started":1}
W20260807 11:17:32.897238 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:32.887632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38635 (local address 127.2.223.190:41473)
0807 11:17:32.887886 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:32.887893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:32.888242 (+ 349us) server_negotiation.cc:407] Connection header received
0807 11:17:32.888435 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:32.888447 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:32.888525 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:32.888631 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:32.889467 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.890229 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.891513 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:32.891951 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:32.892084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:32.896957 (+ 4873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:38635: BlockingRecv error: recv got EOF from 127.2.223.129:38635 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:33.280077 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:33.270547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:49723)
0807 11:17:33.271115 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:33.271131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:33.271249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:33.271545 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:33.271548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:33.271940 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:33.272132 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.272141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.273244 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.273247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.274173 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.274184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.275034 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.275038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.275230 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.275800 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:33.275823 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:33.279700 (+ 3877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":304,"threads_started":1}
W20260807 11:17:33.280251 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:33.270595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49723 (local address 127.2.223.190:41473)
0807 11:17:33.270784 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:33.270788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:33.271207 (+ 419us) server_negotiation.cc:407] Connection header received
0807 11:17:33.271349 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:33.271353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:33.271415 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:33.271497 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:33.272321 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.273103 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.274310 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.274887 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.274999 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.279941 (+ 4942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:49723: BlockingRecv error: recv got EOF from 127.2.223.130:49723 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:33.771564 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:33.761988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:48277)
0807 11:17:33.762583 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:33.762603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:33.762765 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:33.763145 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:33.763152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:33.763585 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:33.763770 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.763779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.764825 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.764831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.765749 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.765760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.766471 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.766477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.766696 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.767252 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:33.767270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:33.771187 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":342,"threads_started":1}
W20260807 11:17:33.771659 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:33.762214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:48277 (local address 127.2.223.190:41473)
0807 11:17:33.762438 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:33.762444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:33.762695 (+ 251us) server_negotiation.cc:407] Connection header received
0807 11:17:33.762899 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:33.762904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:33.762990 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:33.763105 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:33.763923 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.764689 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.765893 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.766294 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.766461 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.771391 (+ 4930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:48277: BlockingRecv error: recv got EOF from 127.2.223.131:48277 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:33 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:33.906814 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:33.898077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:33341)
0807 11:17:33.898636 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:33.898653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:33.898768 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:33.899055 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:33.899058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:33.899408 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:33.899592 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.899600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.900895 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.900899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.901868 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:33.901879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.902626 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.902630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:33.902758 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.903338 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:33.903354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:33.906578 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
W20260807 11:17:33.906999 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:33.898098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:33341 (local address 127.2.223.190:41473)
0807 11:17:33.898333 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:33.898343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:33.898724 (+ 381us) server_negotiation.cc:407] Connection header received
0807 11:17:33.898869 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:33.898873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:33.898950 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:33.899061 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:33.899696 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.900758 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.901990 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:33.902421 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:33.902594 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:33.906768 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:33341: BlockingRecv error: recv got EOF from 127.2.223.129:33341 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:34.290241 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:34.281100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:53791)
0807 11:17:34.281605 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:34.281622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:34.281737 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:34.282078 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:34.282083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:34.282472 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:34.282790 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.282805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.284157 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.284163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.285101 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.285113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.285743 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.285746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.285862 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.286404 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:34.286421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:34.289881 (+ 3460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":289,"threads_started":1}
W20260807 11:17:34.290613 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:34.281116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:53791 (local address 127.2.223.190:41473)
0807 11:17:34.281330 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:34.281339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:34.281701 (+ 362us) server_negotiation.cc:407] Connection header received
0807 11:17:34.281875 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:34.281879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:34.281946 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:34.282034 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:34.282983 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.283991 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.285224 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.285625 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.285714 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.290106 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:53791: BlockingRecv error: recv got EOF from 127.2.223.130:53791 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:34.782248 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:34.772643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:55119)
0807 11:17:34.773163 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:34.773184 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:34.773349 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:34.773715 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:34.773719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:34.774112 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:34.774287 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.774296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.775446 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.775458 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.776545 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.776558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.777306 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.777312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.777495 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.778124 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:34.778142 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:34.781856 (+ 3714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":270,"threads_started":1}
W20260807 11:17:34.782378 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:34.772730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:55119 (local address 127.2.223.190:41473)
0807 11:17:34.772972 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:34.772979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:34.773278 (+ 299us) server_negotiation.cc:407] Connection header received
0807 11:17:34.773475 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:34.773480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:34.773572 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:34.773679 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:34.774445 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.775299 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.776706 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.777152 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.777291 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.782112 (+ 4821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:55119: BlockingRecv error: recv got EOF from 127.2.223.131:55119 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:34 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:34.917291 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:34.907793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:55601)
0807 11:17:34.908319 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:34.908337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:34.908455 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:34.908785 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:34.908788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:34.909216 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:34.909395 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.909405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.910459 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.910464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.911521 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:34.911542 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.912230 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.912234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:34.912391 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.912998 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:34.913016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:34.916896 (+ 3880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
W20260807 11:17:34.917363 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:34.907936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:55601 (local address 127.2.223.190:41473)
0807 11:17:34.908222 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:34.908229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:34.908422 (+ 193us) server_negotiation.cc:407] Connection header received
0807 11:17:34.908568 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:34.908572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:34.908645 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:34.908727 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:34.909570 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.910313 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.911668 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:34.912081 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:34.912194 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:34.917107 (+ 4913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:55601: BlockingRecv error: recv got EOF from 127.2.223.129:55601 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:35.300964 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:35.291318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:54121)
0807 11:17:35.291902 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:35.291919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:35.292109 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:35.292488 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:35.292493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:35.292912 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:35.293084 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.293094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.294126 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.294133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.295168 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.295180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.295901 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.295907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.296098 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.296661 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:35.296678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:35.300594 (+ 3916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
W20260807 11:17:35.301043 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:35.291462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:54121 (local address 127.2.223.190:41473)
0807 11:17:35.291729 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:35.291738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:35.292043 (+ 305us) server_negotiation.cc:407] Connection header received
0807 11:17:35.292250 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:35.292257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:35.292330 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:35.292439 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:35.293238 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.293981 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.295327 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.295756 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.295890 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.300796 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:54121: BlockingRecv error: recv got EOF from 127.2.223.130:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:35.792613 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:35.783283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:54073)
0807 11:17:35.783903 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:35.783923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:35.784037 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:35.784393 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:35.784396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:35.784789 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:35.784963 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.784971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.785958 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.785962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.787159 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.787172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.787911 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.787914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.788050 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.788700 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:35.788717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:35.792375 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":355,"threads_started":1}
W20260807 11:17:35.792817 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:35.783437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:54073 (local address 127.2.223.190:41473)
0807 11:17:35.783695 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:35.783704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:35.784018 (+ 314us) server_negotiation.cc:407] Connection header received
0807 11:17:35.784135 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:35.784139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:35.784221 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:35.784305 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:35.785079 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.785818 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.787316 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.787765 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.787891 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.792564 (+ 4673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:54073: BlockingRecv error: recv got EOF from 127.2.223.131:54073 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:35 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:35.930408 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:35.918332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:44997)
0807 11:17:35.918982 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:35.919001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:35.919126 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:35.919460 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:35.919464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:35.919893 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:35.920133 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.920143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.921152 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.921155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.922093 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:35.922103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.923071 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.923080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:35.923268 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.926658 (+ 3390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:35.926676 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:35.930172 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":367,"threads_started":1}
W20260807 11:17:35.930652 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:35.918430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:44997 (local address 127.2.223.190:41473)
0807 11:17:35.918609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:35.918613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:35.919075 (+ 462us) server_negotiation.cc:407] Connection header received
0807 11:17:35.919228 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:35.919231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:35.919295 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:35.919378 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:35.920283 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.921016 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.922268 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:35.922887 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:35.923029 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:35.930375 (+ 7346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:44997: BlockingRecv error: recv got EOF from 127.2.223.129:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:36.311198 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:36.301825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:57689)
0807 11:17:36.302382 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:36.302399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:36.302576 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:36.302934 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:36.302939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:36.303448 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:36.303601 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.303610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.304676 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.304682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.305678 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.305690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.306441 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.306446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.306673 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.307310 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:36.307328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:36.310856 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":316,"threads_started":1}
W20260807 11:17:36.311316 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:36.302075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:57689 (local address 127.2.223.190:41473)
0807 11:17:36.302303 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:36.302309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:36.302506 (+ 197us) server_negotiation.cc:407] Connection header received
0807 11:17:36.302705 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:36.302710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:36.302787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:36.302900 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:36.303745 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.304528 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.305830 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.306292 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.306428 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.311055 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:57689: BlockingRecv error: recv got EOF from 127.2.223.130:57689 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:36.802596 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:36.793686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:52613)
0807 11:17:36.794170 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:36.794189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:36.794336 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:36.794700 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:36.794706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:36.795133 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:36.795284 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.795293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.796234 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.796240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.797149 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.797160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.797912 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.797917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.798091 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.798783 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:36.798801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:36.802212 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
W20260807 11:17:36.802711 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:36.793719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:52613 (local address 127.2.223.190:41473)
0807 11:17:36.793924 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:36.793931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:36.794275 (+ 344us) server_negotiation.cc:407] Connection header received
0807 11:17:36.794450 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:36.794455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:36.794560 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:36.794659 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:36.795417 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.796109 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.797285 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.797768 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.797901 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.802434 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:52613: BlockingRecv error: recv got EOF from 127.2.223.131:52613 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:36 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:36.940470 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:36.931466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:44883)
0807 11:17:36.932018 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:36.932034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:36.932150 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:36.932446 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:36.932448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:36.932857 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:36.933001 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.933009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.934018 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.934021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.935188 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:36.935201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.936008 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.936014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:36.936194 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.936734 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:36.936750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:36.940144 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":317,"threads_started":1}
W20260807 11:17:36.940608 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:36.931609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:44883 (local address 127.2.223.190:41473)
0807 11:17:36.931805 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:36.931810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:36.932129 (+ 319us) server_negotiation.cc:407] Connection header received
0807 11:17:36.932250 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:36.932254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:36.932318 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:36.932429 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:36.933123 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.933879 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.935329 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:36.935846 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:36.935988 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:36.940357 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:44883: BlockingRecv error: recv got EOF from 127.2.223.129:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:37.322464 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:37.312420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:48555)
0807 11:17:37.312974 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:37.313018 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:37.313151 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:37.313531 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:37.313534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:37.313951 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:37.314143 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.314151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.315149 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.315153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.316104 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.316116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.316913 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.316919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.317107 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.317870 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:37.317891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:37.322104 (+ 4213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
W20260807 11:17:37.322547 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:37.312498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48555 (local address 127.2.223.190:41473)
0807 11:17:37.312728 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:37.312732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:37.313111 (+ 379us) server_negotiation.cc:407] Connection header received
0807 11:17:37.313325 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:37.313329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:37.313397 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:37.313483 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:37.314284 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.315014 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.316262 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.316722 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.316823 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.322300 (+ 5477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:48555: BlockingRecv error: recv got EOF from 127.2.223.130:48555 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:37.812861 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:37.803582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:51737)
0807 11:17:37.804124 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:37.804146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:37.804316 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:37.804686 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:37.804691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:37.805152 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:37.805322 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.805331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.806382 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.806389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.807354 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.807365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.808136 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.808142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.808317 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.808944 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:37.808966 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:37.812631 (+ 3665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":307,"threads_started":1}
W20260807 11:17:37.813058 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:37.803657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:51737 (local address 127.2.223.190:41473)
0807 11:17:37.803944 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:37.803953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:37.804240 (+ 287us) server_negotiation.cc:407] Connection header received
0807 11:17:37.804447 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:37.804453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:37.804541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:37.804647 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:37.805471 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.806243 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.807499 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.807958 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.808094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.812819 (+ 4725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:51737: BlockingRecv error: recv got EOF from 127.2.223.131:51737 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:37 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:37.951753 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:37.941493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:46895)
0807 11:17:37.942172 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:37.942188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:37.942356 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:37.942748 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:37.942754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:37.943332 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:37.943532 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.943542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.944950 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.944954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.945919 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:37.945931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.946863 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.946868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:37.946995 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.947623 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:37.947641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:37.951450 (+ 3809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":281,"threads_started":1}
W20260807 11:17:37.951838 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:37.941697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:46895 (local address 127.2.223.190:41473)
0807 11:17:37.941976 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:37.941982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:37.942285 (+ 303us) server_negotiation.cc:407] Connection header received
0807 11:17:37.942521 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:37.942525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:37.942587 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:37.942693 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:37.943725 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.944820 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.946085 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:37.946690 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:37.946794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:37.951639 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:46895: BlockingRecv error: recv got EOF from 127.2.223.129:46895 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:38.333012 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:38.323622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:47383)
0807 11:17:38.324243 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:38.324260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:38.324395 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:38.324709 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:38.324712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:38.325150 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:38.325366 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.325375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.326426 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.326429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.327445 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.327457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.328259 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.328265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.328441 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.329094 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:38.329124 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:38.332639 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
W20260807 11:17:38.333144 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:38.323716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:47383 (local address 127.2.223.190:41473)
0807 11:17:38.323916 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:38.323921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:38.324381 (+ 460us) server_negotiation.cc:407] Connection header received
0807 11:17:38.324489 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:38.324493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:38.324565 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:38.324661 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:38.325504 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.326307 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.327607 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.328085 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.328246 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.332852 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:47383: BlockingRecv error: recv got EOF from 127.2.223.130:47383 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:38.823523 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:38.814054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:36069)
0807 11:17:38.814789 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:38.814809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:38.814943 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:38.815324 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:38.815327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:38.815764 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:38.815982 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.815991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.816989 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.816993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.817972 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.817984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.818749 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.818753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.818867 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.819565 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:38.819584 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:38.823159 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":399,"threads_started":1}
W20260807 11:17:38.823635 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:38.814178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:36069 (local address 127.2.223.190:41473)
0807 11:17:38.814364 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:38.814368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:38.815025 (+ 657us) server_negotiation.cc:407] Connection header received
0807 11:17:38.815115 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:38.815119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:38.815194 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:38.815279 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:38.816129 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.816876 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.818123 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.818611 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.818705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.823353 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:36069: BlockingRecv error: recv got EOF from 127.2.223.131:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:38 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:38.962273 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:38.952879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:42355)
0807 11:17:38.953462 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:38.953478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:38.953657 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:38.954011 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:38.954015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:38.954496 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:38.954733 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.954742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.955835 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.955842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.956779 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:38.956790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.957597 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.957606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:38.957772 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.958347 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:38.958364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:38.962002 (+ 3638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":284,"threads_started":1}
W20260807 11:17:38.962422 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:38.953142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:42355 (local address 127.2.223.190:41473)
0807 11:17:38.953378 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:38.953387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:38.953577 (+ 190us) server_negotiation.cc:407] Connection header received
0807 11:17:38.953789 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:38.953800 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:38.953864 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:38.953973 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:38.954905 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.955691 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.956927 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:38.957375 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:38.957511 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:38.962200 (+ 4689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:42355: BlockingRecv error: recv got EOF from 127.2.223.129:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:39.344027 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:39.334049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:53397)
0807 11:17:39.334691 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:39.334710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:39.334877 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:39.335266 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:39.335271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:39.335711 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:39.335892 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.335900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.337079 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.337085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.338067 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.338079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.338902 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.338908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.339073 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.339683 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:39.339703 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:39.343592 (+ 3889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":334,"threads_started":1}
W20260807 11:17:39.344132 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:39.334215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:53397 (local address 127.2.223.190:41473)
0807 11:17:39.334468 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:39.334497 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:39.334799 (+ 302us) server_negotiation.cc:407] Connection header received
0807 11:17:39.335004 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:39.335009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:39.335109 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:39.335215 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:39.336038 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.336929 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.338207 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.338706 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.338855 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.343880 (+ 5025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:53397: BlockingRecv error: recv got EOF from 127.2.223.130:53397 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:39.834733 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:39.824704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:48781)
0807 11:17:39.825252 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:39.825280 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:39.825433 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:39.825735 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:39.825738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:39.826216 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:39.826459 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.826471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.827937 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.827941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.828870 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.828882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.829578 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.829583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.829699 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.830358 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:39.830376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:39.834317 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
W20260807 11:17:39.834860 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:39.824832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:48781 (local address 127.2.223.190:41473)
0807 11:17:39.825100 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:39.825106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:39.825390 (+ 284us) server_negotiation.cc:407] Connection header received
0807 11:17:39.825537 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:39.825541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:39.825603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:39.825685 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:39.826646 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.827796 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.829000 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.829411 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.829509 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.834582 (+ 5073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:48781: BlockingRecv error: recv got EOF from 127.2.223.131:48781 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:39 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:39.972880 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:39.963307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:40241)
0807 11:17:39.963847 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:39.963864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:39.963978 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:39.964550 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:39.964556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:39.964987 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:39.965170 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.965180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.966184 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.966188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.967227 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:39.967240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.967937 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.967949 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:39.968101 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.968733 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:39.968752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:39.972563 (+ 3811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
W20260807 11:17:39.973057 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:39.963418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:40241 (local address 127.2.223.190:41473)
0807 11:17:39.963615 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:39.963620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:39.963941 (+ 321us) server_negotiation.cc:407] Connection header received
0807 11:17:39.964263 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:39.964267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:39.964352 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:39.964456 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:39.965314 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.966041 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.967362 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:39.967770 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:39.967866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:39.972784 (+ 4918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:40241: BlockingRecv error: recv got EOF from 127.2.223.129:40241 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:40.354347 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:40.345121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:58463)
0807 11:17:40.345847 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:40.345864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:40.345987 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:40.346277 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:40.346280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:40.346755 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:40.346914 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.346924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.347946 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.347950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.348931 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.348943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.349665 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.349670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.349803 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.350540 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:40.350583 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:40.354128 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":429,"threads_started":1}
W20260807 11:17:40.354563 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:40.345167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:58463 (local address 127.2.223.190:41473)
0807 11:17:40.345418 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:40.345427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:40.345953 (+ 526us) server_negotiation.cc:407] Connection header received
0807 11:17:40.346090 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:40.346094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:40.346161 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:40.346234 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:40.347042 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.347822 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.349080 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.349495 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.349613 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.354290 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:58463: BlockingRecv error: recv got EOF from 127.2.223.130:58463 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:40.846407 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:40.835817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:45077)
0807 11:17:40.836429 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:40.836451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:40.836565 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:40.836867 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:40.836870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:40.837320 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:40.837483 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.837493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.838506 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.838510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.840192 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.840209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.841267 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.841273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.841469 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.842254 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:40.842272 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:40.846071 (+ 3799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":348,"threads_started":1}
W20260807 11:17:40.846465 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:40.835886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:45077 (local address 127.2.223.190:41473)
0807 11:17:40.836128 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:40.836132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:40.836525 (+ 393us) server_negotiation.cc:407] Connection header received
0807 11:17:40.836661 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:40.836665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:40.836727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:40.836810 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:40.837615 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.838347 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.840347 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.841097 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.841244 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.846240 (+ 4996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:45077: BlockingRecv error: recv got EOF from 127.2.223.131:45077 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:40 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:40.987334 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:40.973982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:56139)
0807 11:17:40.974610 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:40.974628 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:40.974793 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:40.975199 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:40.975204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:40.975654 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:40.975808 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.975817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.977402 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.977407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.978684 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:40.978697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.979426 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.979431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:40.979546 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.983376 (+ 3830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:40.983395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:40.987066 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":277,"threads_started":1}
W20260807 11:17:40.987707 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:40.974076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:56139 (local address 127.2.223.190:41473)
0807 11:17:40.974335 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:40.974341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:40.974713 (+ 372us) server_negotiation.cc:407] Connection header received
0807 11:17:40.974964 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:40.974974 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:40.975058 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:40.975163 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:40.976004 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.977253 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.978817 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:40.979283 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:40.979383 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:40.987312 (+ 7929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:56139: BlockingRecv error: recv got EOF from 127.2.223.129:56139 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:41.364809 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:41.355389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:43923)
0807 11:17:41.355919 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:41.355935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:41.356104 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:41.356462 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:41.356467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:41.356868 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:41.357020 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.357031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.358107 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.358112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.359208 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.359225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.360172 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.360181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.360352 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:41.361054 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:41.361072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:41.364563 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":287,"threads_started":1}
W20260807 11:17:41.364984 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:41.355490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43923 (local address 127.2.223.190:41473)
0807 11:17:41.355734 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:41.355743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:41.356034 (+ 291us) server_negotiation.cc:407] Connection header received
0807 11:17:41.356230 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:41.356236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:41.356317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:41.356423 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:41.357179 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.357951 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.359367 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.359986 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.360158 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:41.364750 (+ 4592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:43923: BlockingRecv error: recv got EOF from 127.2.223.130:43923 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:41.857205 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:41.847713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:46935)
0807 11:17:41.848284 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:41.848312 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:41.848428 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:41.848809 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:41.848814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:41.849262 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:41.849435 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.849445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.850543 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.850549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.851576 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.851589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.852347 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.852353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.852538 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:41.853206 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:41.853223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:41.856952 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":284,"threads_started":1}
W20260807 11:17:41.857450 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:41.847828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:46935 (local address 127.2.223.190:41473)
0807 11:17:41.848040 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:41.848046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:41.848382 (+ 336us) server_negotiation.cc:407] Connection header received
0807 11:17:41.848584 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:41.848588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:41.848654 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:41.848767 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:41.849595 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.850370 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.851724 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.852195 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.852326 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:41.857193 (+ 4867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:46935: BlockingRecv error: recv got EOF from 127.2.223.131:46935 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:41 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:42.000720 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:41.988397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:48579)
0807 11:17:41.989367 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:41.989392 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:41.989572 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:41.990022 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:41.990029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:41.990718 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:41.990989 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.991005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.992666 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.992672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.994280 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:41.994295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.995482 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.995487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:41.995647 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:41.996361 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:41.996379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:42.000478 (+ 4099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":623,"threads_started":1}
W20260807 11:17:42.000928 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:41.988501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:48579 (local address 127.2.223.190:41473)
0807 11:17:41.988875 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:41.988885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:41.989507 (+ 622us) server_negotiation.cc:407] Connection header received
0807 11:17:41.989730 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:41.989735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:41.989835 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:41.989955 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:41.991255 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.992450 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.994472 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:41.995336 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:41.995440 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:42.000718 (+ 5278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:48579: BlockingRecv error: recv got EOF from 127.2.223.129:48579 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 11:17:42 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:42 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:42.375742 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:42.365994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:43469)
0807 11:17:42.366597 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:42.366615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:42.366742 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:42.367082 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:42.367086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:42.367560 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:42.367781 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:42.367792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.368834 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.368838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:42.369781 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:42.369793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.370509 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.370514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:42.370647 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:42.371468 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:42.371488 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:42.375385 (+ 3897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":311,"threads_started":1}
W20260807 11:17:42.375871 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:42.366061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43469 (local address 127.2.223.190:41473)
0807 11:17:42.366246 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:42.366250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:42.366695 (+ 445us) server_negotiation.cc:407] Connection header received
0807 11:17:42.366844 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:42.366848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:42.366909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:42.366987 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:42.367921 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.368712 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.369911 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.370345 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.370435 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:42.375582 (+ 5147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:43469: BlockingRecv error: recv got EOF from 127.2.223.130:43469 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 11:17:42 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:42 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:42.867815 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:42.858311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:35265)
0807 11:17:42.858882 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:42.858908 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:42.859037 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:42.859390 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:42.859394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:42.859824 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:42.860029 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:42.860038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.861029 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.861035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:42.862198 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:42.862212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.863001 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.863005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:42.863142 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:42.863826 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:42.863850 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:42.867590 (+ 3740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":343,"threads_started":1}
W20260807 11:17:42.867982 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:42.858415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:35265 (local address 127.2.223.190:41473)
0807 11:17:42.858609 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:42.858613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:42.859003 (+ 390us) server_negotiation.cc:407] Connection header received
0807 11:17:42.859143 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:42.859146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:42.859213 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:42.859299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:42.860162 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.860891 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.862348 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:42.862856 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:42.862960 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:42.867763 (+ 4803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:35265: BlockingRecv error: recv got EOF from 127.2.223.131:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:43.010967 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:43.001648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:48199)
0807 11:17:43.002163 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:43.002180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:43.002326 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:43.002710 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:43.002713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:43.003093 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:43.003269 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.003277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.004273 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.004277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.005231 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.005242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.006064 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.006068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.006202 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.006964 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:43.006982 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:43.010708 (+ 3726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
W20260807 11:17:43.011157 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:43.001704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:48199 (local address 127.2.223.190:41473)
0807 11:17:43.001933 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:43.001942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:43.002290 (+ 348us) server_negotiation.cc:407] Connection header received
0807 11:17:43.002426 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:43.002430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:43.002579 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:43.002663 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:43.003400 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.004154 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.005383 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.005905 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.006035 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.010903 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:48199: BlockingRecv error: recv got EOF from 127.2.223.129:48199 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:43.386927 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:43.376849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:47589)
0807 11:17:43.377555 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:43.377580 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:43.377748 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:43.378162 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:43.378169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:43.378770 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:43.379076 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.379090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.380205 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.380209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.381180 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.381191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.381916 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.381920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.382092 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.382823 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:43.382842 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:43.386657 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":351,"threads_started":1}
W20260807 11:17:43.387020 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:43.376883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:47589 (local address 127.2.223.190:41473)
0807 11:17:43.377100 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:43.377109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:43.377678 (+ 569us) server_negotiation.cc:407] Connection header received
0807 11:17:43.377882 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:43.377887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:43.377950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:43.378068 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:43.379255 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.380041 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.381306 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.381750 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.381859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.386826 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:47589: BlockingRecv error: recv got EOF from 127.2.223.130:47589 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:43 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:43.878527 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:43.868822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:37815)
0807 11:17:43.869350 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:43.869373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:43.869493 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:43.869827 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:43.869831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:43.870306 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:43.870569 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.870591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.871663 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.871667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.872692 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:43.872703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.873382 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.873387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:43.873539 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.874244 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:43.874264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:43.878210 (+ 3946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
W20260807 11:17:43.878752 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:43.868923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:37815 (local address 127.2.223.190:41473)
0807 11:17:43.869179 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:43.869184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:43.869452 (+ 268us) server_negotiation.cc:407] Connection header received
0807 11:17:43.869615 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:43.869619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:43.869696 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:43.869776 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:43.870710 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.871514 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.872810 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:43.873216 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:43.873303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:43.878413 (+ 5110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:37815: BlockingRecv error: recv got EOF from 127.2.223.131:37815 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:44.021041 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:44.011966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:36083)
0807 11:17:44.012515 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:44.012532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:44.012659 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:44.012982 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:44.012985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:44.013412 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:44.013594 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.013603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.014625 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.014629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.015492 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.015502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.016341 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.016347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.016568 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.017239 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:44.017258 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:44.020819 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
W20260807 11:17:44.021361 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:44.012329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:36083 (local address 127.2.223.190:41473)
0807 11:17:44.012545 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:44.012549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:44.012600 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:17:44.012769 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:44.012773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:44.012840 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:44.012936 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:44.013748 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.014499 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.015646 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.016154 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.016330 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.021058 (+ 4728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:36083: BlockingRecv error: recv got EOF from 127.2.223.129:36083 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:44.397410 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:44.387952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:45743)
0807 11:17:44.388559 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:44.388576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:44.388750 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:44.389161 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:44.389166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:44.389603 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:44.389752 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.389761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.390837 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.390842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.391799 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.391810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.392608 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.392614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.392805 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.393457 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:44.393475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:44.397183 (+ 3708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":371,"threads_started":1}
W20260807 11:17:44.397773 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:44.388160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:45743 (local address 127.2.223.190:41473)
0807 11:17:44.388420 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:44.388426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:44.388675 (+ 249us) server_negotiation.cc:407] Connection header received
0807 11:17:44.388876 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:44.388880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:44.388950 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:44.389113 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:44.389901 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.390700 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.391976 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.392445 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.392591 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.397409 (+ 4818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:45743: BlockingRecv error: recv got EOF from 127.2.223.130:45743 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:44 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.131@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:44.889204 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:44.879593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.131:47397)
0807 11:17:44.880102 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:44.880122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:44.880285 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:44.880659 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:44.880664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:44.881159 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:44.881344 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.881353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.882649 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.882653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.883748 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:44.883760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.884509 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.884515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:44.884696 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.885374 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:44.885408 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:44.888820 (+ 3412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
W20260807 11:17:44.889310 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:44.879652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:47397 (local address 127.2.223.190:41473)
0807 11:17:44.879885 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:44.879892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:44.880210 (+ 318us) server_negotiation.cc:407] Connection header received
0807 11:17:44.880432 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:44.880437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:44.880513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:44.880623 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:44.881486 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.882423 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.883934 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:44.884363 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:44.884495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:44.889038 (+ 4543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.131:47397: BlockingRecv error: recv got EOF from 127.2.223.131:47397 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 11:17:45 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:45 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.129@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:17:45.032159 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:45.022324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.129:51599)
0807 11:17:45.022978 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:45.022995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:45.023118 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:45.023537 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:45.023542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:45.024119 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:45.024367 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:45.024382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.025682 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.025685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:45.026736 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:45.026748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.027449 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.027453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:45.027606 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.028253 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:45.028271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:45.031914 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":295,"threads_started":1}
W20260807 11:17:45.032496 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:45.022597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:51599 (local address 127.2.223.190:41473)
0807 11:17:45.023016 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:45.023022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:45.023086 (+ 64us) server_negotiation.cc:407] Connection header received
0807 11:17:45.023284 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:45.023290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:45.023373 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:45.023455 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:45.024538 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.025514 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.026864 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.027309 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.027402 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.032172 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.129:51599: BlockingRecv error: recv got EOF from 127.2.223.129:51599 (error 108)
Metrics: {"server-negotiator.queue_time_us":268}
I20260807 11:17:45.397111 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12271
Aug 07 11:17:45 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:17:45 dist-test-slave-5l3k krb5kdc[11662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.223.130@KRBTEST.COM for oryx/127.2.223.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:17:45.409070 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:45.398898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:39913 (local address 127.2.223.190:41473)
0807 11:17:45.399080 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:45.399086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:45.399620 (+ 534us) server_negotiation.cc:407] Connection header received
0807 11:17:45.399828 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:45.399834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:45.399933 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:45.400051 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:45.400949 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.401780 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.403217 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.403779 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.403880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.408875 (+ 4995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.223.130:39913: BlockingRecv error: recv got EOF from 127.2.223.130:39913 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:17:45.409080 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:45.398798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41473 (local address 127.2.223.130:39913)
0807 11:17:45.399497 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:45.399519 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:45.399678 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:45.400091 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:45.400095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:45.400518 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:45.400790 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:45.400803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.402009 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.402014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:45.403066 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:45.403079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.403934 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.403941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:45.404135 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.404797 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:45.404825 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:45.408693 (+ 3868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.223.190:41473: Server oryx/127.2.223.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":424,"threads_started":1}
I20260807 11:17:45.411911 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:45.398330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.2.223.129:33491)
0807 11:17:45.399110 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:45.399121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:45.399788 (+ 667us) server_negotiation.cc:407] Connection header received
0807 11:17:45.399917 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:45.399926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:45.400325 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:45.400494 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:45.402143 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.403421 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.405047 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.405641 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.405747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.408618 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:45.408634 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:45.408654 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:45.408713 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:45.411429 (+ 2716us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:17:45.411700 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55414: Request ticket server kudu/127.2.223.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
W20260807 11:17:45.412037 12758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55414: Request ticket server kudu/127.2.223.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:17:45.412050 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12271: Not authorized: Client connection negotiation failed: client connection to 127.2.223.129:33491: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:17:45.412197 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12271
I20260807 11:17:45.434664 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12426
I20260807 11:17:45.447813 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:45.435811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.2.223.130:33601)
0807 11:17:45.436450 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:45.436459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:45.436492 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:17:45.436564 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:45.436572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:45.436914 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:45.437158 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:45.438392 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.439910 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.441143 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.441706 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.441805 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.444346 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:45.444355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:45.444369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:45.444415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:45.447248 (+ 2833us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:17:45.447535 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39100: Request ticket server kudu/127.2.223.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
W20260807 11:17:45.447923 12761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39100: Request ticket server kudu/127.2.223.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:17:45.447999 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12426: Not authorized: Client connection negotiation failed: client connection to 127.2.223.130:33601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:17:45.448254 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12426
I20260807 11:17:45.469936 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12572
I20260807 11:17:45.483304 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:45.471236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.2.223.131:37603)
0807 11:17:45.471786 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:45.471793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:45.471819 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:17:45.471889 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:45.471896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:45.472131 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:45.472376 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:45.473800 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.475077 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.476425 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:45.477008 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:45.477179 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:45.479788 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:45.479797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:45.479812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:45.479853 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:45.482461 (+ 2608us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:17:45.482897 (+ 436us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42642: Request ticket server kudu/127.2.223.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
W20260807 11:17:45.483292 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12572: Not authorized: Client connection negotiation failed: client connection to 127.2.223.131:37603: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.223.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:17:45.483424 12762 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42642: Request ticket server kudu/127.2.223.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:17:45.483475 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12572
I20260807 11:17:45.505172 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12200
I20260807 11:17:45.506805 12264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:45.630724 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12200
2026-08-07T11:17:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21860 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T11:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:45Z Disabled control of system clock
I20260807 11:17:45.730912 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42399
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:42399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 11:17:46.078393 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:46.078823 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:46.079037 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:46.086921 12767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:46.087069 12767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:46.087165 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:46.087255 12767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:46.087328 12767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:46.097083 12767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:42399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:46.098582 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:46.100545 12767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:46.113020 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:46.113049 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:46.113111 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:46.113979 12767 server_base.cc:1034] running on GCE node
I20260807 11:17:46.114890 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:46.116458 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:46.117739 12767 hybrid_clock.cc:648] HybridClock initialized: now 1786101466117679 us; error 59 us; skew 500 ppm
I20260807 11:17:46.118146 12767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:46.125830 12767 webserver.cc:460] Webserver started at http://127.2.223.190:33475/ using document root <none> and password file <none>
I20260807 11:17:46.126744 12767 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:46.126886 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:46.127249 12767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:46.130157 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "311b9058bd7c437b8876bd98b8b4238e"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "82e9b2e839159cd4a674e2ed9174b66e"
server_key_iv: "4f19650b3c0140abdb817f3e323dd79a"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.130983 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "311b9058bd7c437b8876bd98b8b4238e"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "82e9b2e839159cd4a674e2ed9174b66e"
server_key_iv: "4f19650b3c0140abdb817f3e323dd79a"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.136098 12767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:17:46.139592 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:46.140954 12767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:46.141183 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "311b9058bd7c437b8876bd98b8b4238e"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "82e9b2e839159cd4a674e2ed9174b66e"
server_key_iv: "4f19650b3c0140abdb817f3e323dd79a"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.141449 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:46.185359 12767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:46.186317 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:46.186677 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:46.206147 12767 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:42399
I20260807 11:17:46.206169 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:42399 every 8 connection(s)
I20260807 11:17:46.207669 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:46.208180 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12767
I20260807 11:17:46.208416 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:46.209329 2942 external_mini_cluster.cc:1428] Setting key a8c398c2133fb6fe8c5ec8c7bb5e9c44
I20260807 11:17:46.212642 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:46.224135 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:46.211630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.2.223.190:42399)
0807 11:17:46.212697 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:46.212708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:46.212780 (+ 72us) server_negotiation.cc:407] Connection header received
0807 11:17:46.214570 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:46.214593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:46.215065 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:46.215422 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:46.216102 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.217418 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.219448 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.220121 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.220261 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:46.221680 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:46.221688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:46.221705 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:46.222296 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:46.222391 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:46.222413 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:46.222586 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:46.222797 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 11:17:46.223152 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":408,"threads_started":1}
I20260807 11:17:46.225545 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e: Bootstrap starting.
I20260807 11:17:46.228792 12837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:46.229791 12837 log.cc:824] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:46.232511 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e: No bootstrap required, opened a new log
I20260807 11:17:46.240109 12837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } }
I20260807 11:17:46.240482 12837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:46.240592 12837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311b9058bd7c437b8876bd98b8b4238e, State: Initialized, Role: FOLLOWER
I20260807 11:17:46.241211 12837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } }
I20260807 11:17:46.241477 12837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:46.241623 12837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:46.241813 12837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:46.244037 12837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } }
I20260807 11:17:46.244546 12837 leader_election.cc:302] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311b9058bd7c437b8876bd98b8b4238e; no voters:
I20260807 11:17:46.245203 12837 leader_election.cc:288] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:46.245368 12843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:46.246909 12843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [term 1 LEADER]: Becoming Leader. State: Replica: 311b9058bd7c437b8876bd98b8b4238e, State: Running, Role: LEADER
I20260807 11:17:46.247553 12837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:46.247478 12843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } }
I20260807 11:17:46.250557 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311b9058bd7c437b8876bd98b8b4238e. Latest consensus state: current_term: 1 leader_uuid: "311b9058bd7c437b8876bd98b8b4238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } } }
I20260807 11:17:46.250988 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:46.250836 12844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311b9058bd7c437b8876bd98b8b4238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311b9058bd7c437b8876bd98b8b4238e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42399 } } }
I20260807 11:17:46.251149 12844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:46.252588 12859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:46.257589 12859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:46.267657 12859 catalog_manager.cc:1347] Generated new cluster ID: 6812a206f8174745a0d2f414f78844c5
I20260807 11:17:46.267787 12859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:46.280103 12859 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:46.281432 12859 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:46.294850 12859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311b9058bd7c437b8876bd98b8b4238e: Generated new TSK 0
I20260807 11:17:46.295991 12859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:17:46.313992 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--builtin_ntp_servers=127.2.223.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 11:17:46.657599 12869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:46.657918 12869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:46.658064 12869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:46.665490 12869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:46.665585 12869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:46.665786 12869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:46.675009 12869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:46.676347 12869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:46.678196 12869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:46.691617 12877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:46.691689 12875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:46.691658 12874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:46.692749 12869 server_base.cc:1034] running on GCE node
I20260807 11:17:46.693367 12869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:46.694300 12869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:46.695499 12869 hybrid_clock.cc:648] HybridClock initialized: now 1786101466695431 us; error 55 us; skew 500 ppm
I20260807 11:17:46.695815 12869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:46.698065 12869 webserver.cc:460] Webserver started at http://127.2.223.129:38435/ using document root <none> and password file <none>
I20260807 11:17:46.698828 12869 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:46.698952 12869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:46.699290 12869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:46.702775 12869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "272f51c4043c485bbeb64965b3631f43"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "feb6b7e30b581e4b715988ce7d3ed8e9"
server_key_iv: "3dd73722fcec0f89ee6b34ba25f08ef0"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.703665 12869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "272f51c4043c485bbeb64965b3631f43"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "feb6b7e30b581e4b715988ce7d3ed8e9"
server_key_iv: "3dd73722fcec0f89ee6b34ba25f08ef0"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.709072 12869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:46.712644 12885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:46.714016 12869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:46.714208 12869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "272f51c4043c485bbeb64965b3631f43"
format_stamp: "Formatted at 2026-08-07 11:17:46 on dist-test-slave-5l3k"
server_key: "feb6b7e30b581e4b715988ce7d3ed8e9"
server_key_iv: "3dd73722fcec0f89ee6b34ba25f08ef0"
server_key_version: "encryptionkey@0"
I20260807 11:17:46.714438 12869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:46.740825 12869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:46.741711 12869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:46.742038 12869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:46.743165 12869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:46.744648 12869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:46.744772 12869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:46.744930 12869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:46.745024 12869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:46.781083 12869 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:46783
I20260807 11:17:46.781193 13002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:46783 every 8 connection(s)
I20260807 11:17:46.782459 12869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:46.784157 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 12869
I20260807 11:17:46.784363 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:46.785086 2942 external_mini_cluster.cc:1428] Setting key d49c9dc9217234615b73a2e45714f2c3
I20260807 11:17:46.788594 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--builtin_ntp_servers=127.2.223.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 11:17:46.797044 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:46.785476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:43737 (local address 127.2.223.190:42399)
0807 11:17:46.788072 (+ 2596us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:46.788079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:46.788105 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:17:46.789643 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:46.789649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:46.789749 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:46.789882 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:46.791109 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.791961 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.793792 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.794222 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.794446 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:46.796051 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:46.796057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:46.796060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:46.796080 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:46.796130 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:46.796144 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:46.796255 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:46.796511 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 11:17:46.796619 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2445,"thread_start_us":2313,"threads_started":1}
I20260807 11:17:46.798154 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:46.786061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42399 (local address 127.2.223.129:43737)
0807 11:17:46.787256 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:46.787313 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:46.789343 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:46.790136 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:46.790145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:46.790244 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:46.790920 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:46.790941 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.792157 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.792162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:46.793630 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:46.793645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:46.794569 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:46.794578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:46.794822 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:46.795685 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:46.795688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:46.795729 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:46.795817 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:46.796254 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:46.796269 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:46.796280 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:17:46.796499 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 11:17:46.796895 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":423,"threads_started":1}
I20260807 11:17:46.799878 13003 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42399
I20260807 11:17:46.800235 13003 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:46.801039 13003 heartbeater.cc:511] Master 127.2.223.190:42399 requested a full tablet report, sending...
I20260807 11:17:46.803342 12800 ts_manager.cc:194] Registered new tserver with Master: 272f51c4043c485bbeb64965b3631f43 (127.2.223.129:46783)
I20260807 11:17:46.804831 12800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.129:43737
W20260807 11:17:47.140791 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:47.141099 13007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:47.141268 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:47.148617 13007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:47.148749 13007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:47.148932 13007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:47.157999 13007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:47.159337 13007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:47.161183 13007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:47.174180 13013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:47.174242 13015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:47.174314 13012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:47.175266 13007 server_base.cc:1034] running on GCE node
I20260807 11:17:47.175889 13007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:47.176757 13007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:47.177979 13007 hybrid_clock.cc:648] HybridClock initialized: now 1786101467177936 us; error 39 us; skew 500 ppm
I20260807 11:17:47.178349 13007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:47.180603 13007 webserver.cc:460] Webserver started at http://127.2.223.130:37847/ using document root <none> and password file <none>
I20260807 11:17:47.181303 13007 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:47.181393 13007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:47.181720 13007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:47.184595 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "8bed2be75a29443eb10ff004cbfcf248"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "727939e61d9985dd1910e6ff77732149"
server_key_iv: "cb7d6a510885bb0ee86e88868882a44d"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.185406 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "8bed2be75a29443eb10ff004cbfcf248"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "727939e61d9985dd1910e6ff77732149"
server_key_iv: "cb7d6a510885bb0ee86e88868882a44d"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.190639 13007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 11:17:47.194135 13022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:47.195518 13007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:47.195706 13007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "8bed2be75a29443eb10ff004cbfcf248"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "727939e61d9985dd1910e6ff77732149"
server_key_iv: "cb7d6a510885bb0ee86e88868882a44d"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.195942 13007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:47.215333 13007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:47.216176 13007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:47.216490 13007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:47.217542 13007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:47.219386 13007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:47.219517 13007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:47.219630 13007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:47.219732 13007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:47.254441 13007 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:36085
I20260807 11:17:47.254526 13139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:36085 every 8 connection(s)
I20260807 11:17:47.255789 13007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:47.259534 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13007
I20260807 11:17:47.259701 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:47.260288 2942 external_mini_cluster.cc:1428] Setting key 585313cc37b3aff7333accd55d590b63
I20260807 11:17:47.263506 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--builtin_ntp_servers=127.2.223.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 11:17:47.269672 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.258370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:40021 (local address 127.2.223.190:42399)
0807 11:17:47.258622 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.258629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.260192 (+ 1563us) server_negotiation.cc:407] Connection header received
0807 11:17:47.262416 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.262421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.262526 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.262632 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.263640 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.264832 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.266529 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.267216 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.267359 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.268779 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:47.268787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:47.268795 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:47.268820 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:47.268869 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:47.268888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:47.268982 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:47.269188 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 11:17:47.269292 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:17:47.270653 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.258904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42399 (local address 127.2.223.130:40021)
0807 11:17:47.260007 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:47.260069 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:47.262104 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:47.262830 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:47.262838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:47.262921 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:47.263458 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:47.263475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.264995 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.264999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:47.266345 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:47.266362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.267528 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.267542 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:47.267760 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.268458 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:47.268461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:47.268496 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:47.268580 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:47.268986 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:47.268994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:47.269001 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:17:47.269138 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 11:17:47.269410 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260807 11:17:47.272059 13140 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42399
I20260807 11:17:47.272399 13140 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:47.273326 13140 heartbeater.cc:511] Master 127.2.223.190:42399 requested a full tablet report, sending...
I20260807 11:17:47.274919 12800 ts_manager.cc:194] Registered new tserver with Master: 8bed2be75a29443eb10ff004cbfcf248 (127.2.223.130:36085)
I20260807 11:17:47.275714 12800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.130:40021
W20260807 11:17:47.606806 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:47.607111 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:47.607208 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:47.614614 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:47.614712 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:47.614815 13143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:47.624538 13143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:42399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:47.625785 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:47.627727 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:47.641127 13148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:47.641232 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:47.641278 13149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:47.642755 13143 server_base.cc:1034] running on GCE node
I20260807 11:17:47.643376 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:47.644359 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:47.645584 13143 hybrid_clock.cc:648] HybridClock initialized: now 1786101467645524 us; error 57 us; skew 500 ppm
I20260807 11:17:47.645893 13143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:47.648468 13143 webserver.cc:460] Webserver started at http://127.2.223.131:39549/ using document root <none> and password file <none>
I20260807 11:17:47.649201 13143 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:47.649330 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:47.649650 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:47.652648 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "6640254ce8b9432bb871a836de5127cc"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "7ed1ad187e4fe51020a995d0dbca1026"
server_key_iv: "54d24a387fd859fa8840250875295ce6"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.653469 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "6640254ce8b9432bb871a836de5127cc"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "7ed1ad187e4fe51020a995d0dbca1026"
server_key_iv: "54d24a387fd859fa8840250875295ce6"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.658764 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 11:17:47.662156 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:47.663558 13143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:47.663738 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "6640254ce8b9432bb871a836de5127cc"
format_stamp: "Formatted at 2026-08-07 11:17:47 on dist-test-slave-5l3k"
server_key: "7ed1ad187e4fe51020a995d0dbca1026"
server_key_iv: "54d24a387fd859fa8840250875295ce6"
server_key_version: "encryptionkey@0"
I20260807 11:17:47.663949 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:47.690630 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:47.691591 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:47.691892 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:47.692970 13143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:47.694461 13143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:47.694630 13143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 11:17:47.694761 13143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:47.694859 13143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:47.729626 13143 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:42887
I20260807 11:17:47.729719 13275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:42887 every 8 connection(s)
I20260807 11:17:47.731011 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:47.735260 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13143
I20260807 11:17:47.735416 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:47.735841 2942 external_mini_cluster.cc:1428] Setting key 54fb87325465cf3a0a83bffaf1e03a0c
I20260807 11:17:47.744028 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.733666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:35921 (local address 127.2.223.190:42399)
0807 11:17:47.733879 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.733886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.735287 (+ 1401us) server_negotiation.cc:407] Connection header received
0807 11:17:47.737560 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.737566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.737664 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.737820 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.739192 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.739918 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.741337 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.741737 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.741835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.743184 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:47.743191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:47.743196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:47.743216 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:47.743277 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:47.743295 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:47.743404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:47.743700 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 11:17:47.743785 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:17:47.745078 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.734214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:42399 (local address 127.2.223.131:35921)
0807 11:17:47.735135 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:47.735170 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:47.737295 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:47.738088 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:47.738100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:47.738218 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:47.739044 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:47.739064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.740038 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.740041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:47.741206 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:47.741218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.741922 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.741927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:47.742110 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.742861 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:47.742864 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:47.742913 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:47.742986 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:47.743432 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:47.743442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:47.743448 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 11:17:47.743684 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 11:17:47.743981 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":308,"threads_started":1}
I20260807 11:17:47.746505 13276 heartbeater.cc:347] Connected to a master server at 127.2.223.190:42399
I20260807 11:17:47.746845 13276 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:47.747522 13276 heartbeater.cc:511] Master 127.2.223.190:42399 requested a full tablet report, sending...
I20260807 11:17:47.748847 12800 ts_manager.cc:194] Registered new tserver with Master: 6640254ce8b9432bb871a836de5127cc (127.2.223.131:42887)
I20260807 11:17:47.749924 12800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.131:35921
I20260807 11:17:47.751243 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:17:47.757480 13005 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:47.755861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.2.223.190:42399)
0807 11:17:47.756037 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.756042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.756573 (+ 531us) server_negotiation.cc:407] Connection header received
0807 11:17:47.756702 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.756706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.756772 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.756851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.757207 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33344: BlockingRecv error: recv got EOF from 127.0.0.1:33344 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:17:47.758353 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12869
I20260807 11:17:47.766017 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.759304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.2.223.129:46783)
0807 11:17:47.759877 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.759884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.759910 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:17:47.759964 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.759970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.760186 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.760314 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.760680 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.761684 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.763056 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.763591 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.763689 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.764668 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:47.764676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:47.764689 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:47.765265 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:47.765310 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:47.765328 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:47.765430 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:47.765611 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 11:17:47.765723 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":297,"threads_started":1}
I20260807 11:17:47.767459 12998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:47.848770 13003 heartbeater.cc:503] Master 127.2.223.190:42399 was elected leader, sending a full tablet report...
I20260807 11:17:47.850219 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12869
W20260807 11:17:47.875157 12784 connection.cc:535] server connection from 127.2.223.129:43737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:17:47.875550 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13007
I20260807 11:17:47.883656 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.876479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.2.223.130:36085)
0807 11:17:47.877040 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.877048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.877075 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:47.877145 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.877152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.877375 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.877549 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.878018 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.879091 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.880379 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.880944 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.881097 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.882107 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:47.882116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:47.882130 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:47.882807 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:47.882855 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:47.882875 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:47.883003 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:47.883181 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 11:17:47.883328 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":277,"threads_started":1}
I20260807 11:17:47.885052 13135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:47.968804 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13007
I20260807 11:17:47.989450 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13143
I20260807 11:17:47.998281 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:47.990705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.2.223.131:42887)
0807 11:17:47.991425 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:47.991439 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:47.991472 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:17:47.991537 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:47.991543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:47.991778 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:47.991912 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:47.992405 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.993515 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.994881 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:47.995473 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:47.995580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:47.996778 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:47.996787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:47.996801 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:47.997407 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:47.997454 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:47.997473 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:47.997580 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:47.997711 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 11:17:47.997835 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":382,"threads_started":1}
I20260807 11:17:47.999682 13271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:48.083678 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13143
I20260807 11:17:48.104275 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 12767
I20260807 11:17:48.105654 12831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:48.210317 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 12767
2026-08-07T11:17:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2571 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T11:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:48Z Disabled control of system clock
I20260807 11:17:48.299337 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36969
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 11:17:48.639106 13295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:48.639415 13295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:48.639500 13295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:48.646559 13295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:48.646659 13295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:48.646704 13295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:48.646760 13295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:48.646804 13295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:48.655695 13295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:36969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:48.656944 13295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:48.658785 13295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:48.670970 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:48.670943 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:48.671068 13303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:48.671806 13295 server_base.cc:1034] running on GCE node
I20260807 11:17:48.672641 13295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:48.673995 13295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:48.675194 13295 hybrid_clock.cc:648] HybridClock initialized: now 1786101468675160 us; error 32 us; skew 500 ppm
I20260807 11:17:48.675616 13295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:48.677891 13295 webserver.cc:460] Webserver started at http://127.2.223.190:45477/ using document root <none> and password file <none>
I20260807 11:17:48.678675 13295 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:48.678803 13295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:48.679141 13295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:48.682015 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "b2e717ce84d74c1db89cda318fa9aa32"
format_stamp: "Formatted at 2026-08-07 11:17:48 on dist-test-slave-5l3k"
server_key: "0b0a4fbcb46d3f0420d0a3b0f0ca99d3"
server_key_iv: "a72e3709200a26fca3b7ad406dab132a"
server_key_version: "encryptionkey@0"
I20260807 11:17:48.682891 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "b2e717ce84d74c1db89cda318fa9aa32"
format_stamp: "Formatted at 2026-08-07 11:17:48 on dist-test-slave-5l3k"
server_key: "0b0a4fbcb46d3f0420d0a3b0f0ca99d3"
server_key_iv: "a72e3709200a26fca3b7ad406dab132a"
server_key_version: "encryptionkey@0"
I20260807 11:17:48.688745 13295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:48.692196 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:48.693518 13295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:17:48.693719 13295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "b2e717ce84d74c1db89cda318fa9aa32"
format_stamp: "Formatted at 2026-08-07 11:17:48 on dist-test-slave-5l3k"
server_key: "0b0a4fbcb46d3f0420d0a3b0f0ca99d3"
server_key_iv: "a72e3709200a26fca3b7ad406dab132a"
server_key_version: "encryptionkey@0"
I20260807 11:17:48.693955 13295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:48.699805 13295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:48.700630 13295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:48.700953 13295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:48.721103 13295 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:36969
I20260807 11:17:48.721160 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:36969 every 8 connection(s)
I20260807 11:17:48.722472 13295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:48.726359 13364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:48.726852 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13295
I20260807 11:17:48.727106 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:48.727784 2942 external_mini_cluster.cc:1428] Setting key 212065969e47152e0afa899adae0b3f9
I20260807 11:17:48.737910 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:48.730037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.2.223.190:36969)
0807 11:17:48.731292 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:48.731306 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:48.731355 (+ 49us) server_negotiation.cc:407] Connection header received
0807 11:17:48.733909 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:48.733941 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:48.734530 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:48.734978 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:48.735922 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:48.735929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:48.735945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:48.735960 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:48.736033 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:48.736054 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:48.736267 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:48.736392 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 11:17:48.736698 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":435,"threads_started":1}
I20260807 11:17:48.741748 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32: Bootstrap starting.
I20260807 11:17:48.744606 13364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:48.745668 13364 log.cc:824] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:48.748499 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32: No bootstrap required, opened a new log
I20260807 11:17:48.756098 13364 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } }
I20260807 11:17:48.756444 13364 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:48.756537 13364 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e717ce84d74c1db89cda318fa9aa32, State: Initialized, Role: FOLLOWER
I20260807 11:17:48.757139 13364 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } }
I20260807 11:17:48.757364 13364 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:48.757488 13364 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:48.757659 13364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:48.760076 13364 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } }
I20260807 11:17:48.760637 13364 leader_election.cc:302] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e717ce84d74c1db89cda318fa9aa32; no voters:
I20260807 11:17:48.761327 13364 leader_election.cc:288] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:48.761451 13369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:48.762202 13369 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [term 1 LEADER]: Becoming Leader. State: Replica: b2e717ce84d74c1db89cda318fa9aa32, State: Running, Role: LEADER
I20260807 11:17:48.762776 13369 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } }
I20260807 11:17:48.763494 13364 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:48.765671 13370 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e717ce84d74c1db89cda318fa9aa32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } } }
I20260807 11:17:48.765691 13371 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e717ce84d74c1db89cda318fa9aa32. Latest consensus state: current_term: 1 leader_uuid: "b2e717ce84d74c1db89cda318fa9aa32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e717ce84d74c1db89cda318fa9aa32" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36969 } } }
I20260807 11:17:48.766114 13370 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:48.766111 13371 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:48.768204 13376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:48.773175 13376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:48.784199 13376 catalog_manager.cc:1347] Generated new cluster ID: 75f5f4299cbd4769bb3840ecb8963e80
I20260807 11:17:48.784373 13376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:48.799067 13376 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:48.799556 13376 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:48.811924 13376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e717ce84d74c1db89cda318fa9aa32: Generated new TSK 0
I20260807 11:17:48.812981 13376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:17:48.824342 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--builtin_ntp_servers=127.2.223.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 11:17:49.174748 13389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:49.175024 13389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:49.175177 13389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:49.182649 13389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:49.182775 13389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:49.182889 13389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:49.192265 13389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:49.193625 13389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:49.195575 13389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:49.208958 13394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:49.209049 13396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:49.209106 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:49.210001 13389 server_base.cc:1034] running on GCE node
I20260807 11:17:49.210719 13389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:49.211753 13389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:49.212988 13389 hybrid_clock.cc:648] HybridClock initialized: now 1786101469212957 us; error 47 us; skew 500 ppm
I20260807 11:17:49.213331 13389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:49.215647 13389 webserver.cc:460] Webserver started at http://127.2.223.129:35807/ using document root <none> and password file <none>
I20260807 11:17:49.216353 13389 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:49.216499 13389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:49.216877 13389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:49.219901 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "0e0a60b2bcd14358853683cbc1dc27fc"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "9718060a3edd738ceb09c74dce24ba66"
server_key_iv: "8163aff26665be991405d63883a1c016"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.220723 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "0e0a60b2bcd14358853683cbc1dc27fc"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "9718060a3edd738ceb09c74dce24ba66"
server_key_iv: "8163aff26665be991405d63883a1c016"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.225816 13389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 11:17:49.229622 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.231057 13389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 11:17:49.231241 13389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "0e0a60b2bcd14358853683cbc1dc27fc"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "9718060a3edd738ceb09c74dce24ba66"
server_key_iv: "8163aff26665be991405d63883a1c016"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.231460 13389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:49.237171 13389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:49.237975 13389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:49.238271 13389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:49.239280 13389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:49.240818 13389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:49.240949 13389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.241082 13389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:49.241174 13389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.276293 13389 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:33415
I20260807 11:17:49.276383 13524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:33415 every 8 connection(s)
I20260807 11:17:49.277642 13389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:49.285184 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13389
I20260807 11:17:49.285441 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:49.286106 2942 external_mini_cluster.cc:1428] Setting key bd322c2014f759a6c123ed67e40e904c
I20260807 11:17:49.286382 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:49.280394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:49331 (local address 127.2.223.190:36969)
0807 11:17:49.280843 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:49.280847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:49.282075 (+ 1228us) server_negotiation.cc:407] Connection header received
0807 11:17:49.283893 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:49.283898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:49.283968 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:49.284100 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:49.285502 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:49.285508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:49.285512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:49.285551 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:49.285612 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:49.285627 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:49.285775 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:49.285973 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 11:17:49.286044 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260807 11:17:49.287431 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:49.281027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36969 (local address 127.2.223.129:49331)
0807 11:17:49.281926 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:49.281962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:49.283624 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:49.284351 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:49.284358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:49.284444 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:49.285147 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:49.285149 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:49.285186 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:49.285297 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:49.285735 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:49.285746 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:49.285755 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 11:17:49.286066 (+ 311us) client_negotiation.cc:240] Negotiation successful
0807 11:17:49.286318 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260807 11:17:49.289083 13525 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36969
I20260807 11:17:49.289340 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--builtin_ntp_servers=127.2.223.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 11:17:49.289439 13525 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:49.290154 13525 heartbeater.cc:511] Master 127.2.223.190:36969 requested a full tablet report, sending...
I20260807 11:17:49.292569 13329 ts_manager.cc:194] Registered new tserver with Master: 0e0a60b2bcd14358853683cbc1dc27fc (127.2.223.129:33415)
W20260807 11:17:49.632797 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:49.633068 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:49.633221 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:49.640518 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:49.640609 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:49.640796 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:49.649677 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:49.650983 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:49.652781 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:49.666261 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:49.666364 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:49.666698 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:49.667384 13529 server_base.cc:1034] running on GCE node
I20260807 11:17:49.668068 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:49.669060 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:49.670317 13529 hybrid_clock.cc:648] HybridClock initialized: now 1786101469670263 us; error 48 us; skew 500 ppm
I20260807 11:17:49.670711 13529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:49.673254 13529 webserver.cc:460] Webserver started at http://127.2.223.130:41875/ using document root <none> and password file <none>
I20260807 11:17:49.674013 13529 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:49.674151 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:49.674623 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:49.677608 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "e592293d18a84383aaab07d3767e6b62"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "1d884c37cda0a203e96bb74c9d0e878a"
server_key_iv: "f80ef20f879dd755b1ae15587fe37332"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.678455 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "e592293d18a84383aaab07d3767e6b62"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "1d884c37cda0a203e96bb74c9d0e878a"
server_key_iv: "f80ef20f879dd755b1ae15587fe37332"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.683681 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:17:49.687179 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.688597 13529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:17:49.688786 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "e592293d18a84383aaab07d3767e6b62"
format_stamp: "Formatted at 2026-08-07 11:17:49 on dist-test-slave-5l3k"
server_key: "1d884c37cda0a203e96bb74c9d0e878a"
server_key_iv: "f80ef20f879dd755b1ae15587fe37332"
server_key_version: "encryptionkey@0"
I20260807 11:17:49.689055 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:49.694901 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:49.695863 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:49.696146 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:49.697090 13529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:49.698582 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:49.698670 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.698750 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:49.698804 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:49.735288 13529 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:38439
I20260807 11:17:49.735277 13662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:38439 every 8 connection(s)
I20260807 11:17:49.736783 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:49.740002 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13529
I20260807 11:17:49.740221 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:49.740819 2942 external_mini_cluster.cc:1428] Setting key 37a2661de78a8829c3419d66b724ada0
I20260807 11:17:49.743867 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--builtin_ntp_servers=127.2.223.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 11:17:49.746129 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:49.739750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:35477 (local address 127.2.223.190:36969)
0807 11:17:49.739925 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:49.739930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:49.741761 (+ 1831us) server_negotiation.cc:407] Connection header received
0807 11:17:49.743774 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:49.743780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:49.743847 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:49.743968 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:49.745347 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:49.745354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:49.745359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:49.745377 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:49.745418 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:49.745430 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:49.745555 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:49.745770 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 11:17:49.745844 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:17:49.747131 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:49.740475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36969 (local address 127.2.223.130:35477)
0807 11:17:49.741583 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:49.741644 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:49.743497 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:49.744220 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:49.744228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:49.744320 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:49.745053 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:49.745055 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:49.745095 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:49.745182 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:49.745526 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:49.745537 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:49.745545 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 11:17:49.745857 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 11:17:49.746120 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":347,"threads_started":1}
I20260807 11:17:49.748690 13664 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36969
I20260807 11:17:49.748993 13664 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:49.749644 13664 heartbeater.cc:511] Master 127.2.223.190:36969 requested a full tablet report, sending...
I20260807 11:17:49.751207 13329 ts_manager.cc:194] Registered new tserver with Master: e592293d18a84383aaab07d3767e6b62 (127.2.223.130:38439)
W20260807 11:17:50.090035 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:50.090335 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:50.090430 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:50.098234 13667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:50.098340 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:50.098450 13667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:50.107851 13667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:50.109148 13667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:50.110984 13667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:50.124114 13673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:50.124167 13675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:50.124214 13672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:50.125174 13667 server_base.cc:1034] running on GCE node
I20260807 11:17:50.125738 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:50.126725 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:50.127928 13667 hybrid_clock.cc:648] HybridClock initialized: now 1786101470127889 us; error 34 us; skew 500 ppm
I20260807 11:17:50.128257 13667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:17:50.130569 13667 webserver.cc:460] Webserver started at http://127.2.223.131:46089/ using document root <none> and password file <none>
I20260807 11:17:50.131255 13667 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:50.131377 13667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:50.131728 13667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:50.134723 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "c51e589f3dd94ee79ae12713c67552c7"
format_stamp: "Formatted at 2026-08-07 11:17:50 on dist-test-slave-5l3k"
server_key: "1bc4eade633c673ad06dda0bcffeb1ef"
server_key_iv: "6fe8a1e15c6210d520ae3647f4a91187"
server_key_version: "encryptionkey@0"
I20260807 11:17:50.135695 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "c51e589f3dd94ee79ae12713c67552c7"
format_stamp: "Formatted at 2026-08-07 11:17:50 on dist-test-slave-5l3k"
server_key: "1bc4eade633c673ad06dda0bcffeb1ef"
server_key_iv: "6fe8a1e15c6210d520ae3647f4a91187"
server_key_version: "encryptionkey@0"
I20260807 11:17:50.140928 13667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:50.144464 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:50.146018 13667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:17:50.146234 13667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "c51e589f3dd94ee79ae12713c67552c7"
format_stamp: "Formatted at 2026-08-07 11:17:50 on dist-test-slave-5l3k"
server_key: "1bc4eade633c673ad06dda0bcffeb1ef"
server_key_iv: "6fe8a1e15c6210d520ae3647f4a91187"
server_key_version: "encryptionkey@0"
I20260807 11:17:50.146472 13667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:50.152407 13667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:50.153292 13667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:50.153612 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:50.154723 13667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:50.156351 13667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:50.156452 13667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:50.156636 13667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:50.156733 13667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:50.192934 13667 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:45497
I20260807 11:17:50.193045 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:45497 every 8 connection(s)
I20260807 11:17:50.194428 13667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:50.196012 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13667
I20260807 11:17:50.196225 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:50.196688 2942 external_mini_cluster.cc:1428] Setting key 31eec0f449164d10fa47f021e5d49bc5
I20260807 11:17:50.203097 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:50.197193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:39807 (local address 127.2.223.190:36969)
0807 11:17:50.197454 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:50.197460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:50.199136 (+ 1676us) server_negotiation.cc:407] Connection header received
0807 11:17:50.201109 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:50.201113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:50.201177 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:50.201261 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:50.202376 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:50.202380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:50.202384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:50.202398 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:50.202430 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:50.202441 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:50.202610 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:50.202808 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 11:17:50.202863 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 11:17:50.203959 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:50.197752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36969 (local address 127.2.223.131:39807)
0807 11:17:50.198961 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:50.199002 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:50.200867 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:50.201426 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:50.201434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:50.201527 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:50.202112 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:17:50.202115 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:50.202149 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:17:50.202230 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:50.202586 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:50.202594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:50.202599 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:17:50.202813 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 11:17:50.203008 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":342,"threads_started":1}
I20260807 11:17:50.205313 13802 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36969
I20260807 11:17:50.205629 13802 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:50.206239 13802 heartbeater.cc:511] Master 127.2.223.190:36969 requested a full tablet report, sending...
I20260807 11:17:50.207654 13329 ts_manager.cc:194] Registered new tserver with Master: c51e589f3dd94ee79ae12713c67552c7 (127.2.223.131:45497)
I20260807 11:17:50.212405 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:17:50.218760 13527 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:17:50.216998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.2.223.190:36969)
0807 11:17:50.217282 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:50.217286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:50.217758 (+ 472us) server_negotiation.cc:407] Connection header received
0807 11:17:50.217909 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:50.217913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:50.217965 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:50.218070 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:50.218386 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37756: BlockingRecv error: recv got EOF from 127.0.0.1:37756 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
I20260807 11:17:50.220060 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13389
I20260807 11:17:50.223726 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:50.220981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.2.223.129:33415)
0807 11:17:50.221608 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:50.221614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:50.221641 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:17:50.221699 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:50.221705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:50.221912 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:50.222038 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:50.223016 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:50.223022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:50.223033 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:50.223050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:50.223098 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:50.223111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:50.223216 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:50.223296 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 11:17:50.223403 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":343,"threads_started":1}
I20260807 11:17:50.224980 13520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:50.306946 13525 heartbeater.cc:503] Master 127.2.223.190:36969 was elected leader, sending a full tablet report...
I20260807 11:17:50.308270 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13389
W20260807 11:17:50.325659 13312 connection.cc:535] server connection from 127.2.223.129:49331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:17:50.325999 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13529
I20260807 11:17:50.329483 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:50.326969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.2.223.130:38439)
0807 11:17:50.327487 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:50.327494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:50.327522 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:17:50.327591 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:50.327597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:50.327809 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:50.327929 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:50.328808 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:50.328814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:50.328826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:50.328838 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:50.328879 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:50.328892 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:50.329001 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:50.329072 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 11:17:50.329189 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
I20260807 11:17:50.330785 13658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:50.430508 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13529
I20260807 11:17:50.451257 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13667
I20260807 11:17:50.455355 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:50.452291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.2.223.131:45497)
0807 11:17:50.452952 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:50.452966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:50.452997 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:17:50.453062 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:50.453068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:50.453287 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:50.453407 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:50.454567 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:50.454575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:50.454606 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:17:50.454621 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:50.454667 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:17:50.454682 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:50.454818 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:50.454877 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 11:17:50.454992 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":352,"threads_started":1}
I20260807 11:17:50.456611 13797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:50.538836 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13667
I20260807 11:17:50.559399 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13295
I20260807 11:17:50.560644 13359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:50.665118 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13295
2026-08-07T11:17:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2456 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:50 dist-test-slave-5l3k krb5kdc[13821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:50 dist-test-slave-5l3k krb5kdc[13821](info): set up 2 sockets
Aug 07 11:17:50 dist-test-slave-5l3k krb5kdc[13821](info): commencing operation
krb5kdc: starting...
W20260807 11:17:51.832455 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:52 dist-test-slave-5l3k krb5kdc[13821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:17:52.150832 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45845
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:52.496204 13835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:52.496502 13835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:52.496663 13835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:52.503934 13835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:17:52.504038 13835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:52.504156 13835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:52.504272 13835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:17:52.504333 13835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:17:52.514040 13835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:52.515416 13835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:52.516738 13835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:52.529198 13840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:52.529115 13841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:52.529181 13843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:52.530085 13835 server_base.cc:1034] running on GCE node
I20260807 11:17:52.530989 13835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:52.532382 13835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:52.533607 13835 hybrid_clock.cc:648] HybridClock initialized: now 1786101472533574 us; error 63 us; skew 500 ppm
I20260807 11:17:52.533937 13835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:52 dist-test-slave-5l3k krb5kdc[13821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:52.540438 13835 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:17:52.542021 13835 webserver.cc:460] Webserver started at http://127.2.223.190:33683/ using document root <none> and password file <none>
I20260807 11:17:52.542779 13835 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:52.542930 13835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:52.543268 13835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:52.546123 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "2032bd73292e4cfa8f42dfc1a5e37b24"
format_stamp: "Formatted at 2026-08-07 11:17:52 on dist-test-slave-5l3k"
server_key: "c0ff1fca0ccb0d855a3e426ae27b7e74"
server_key_iv: "0f07a85d31eb80e6500874ae85ebbaf8"
server_key_version: "encryptionkey@0"
I20260807 11:17:52.546880 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "2032bd73292e4cfa8f42dfc1a5e37b24"
format_stamp: "Formatted at 2026-08-07 11:17:52 on dist-test-slave-5l3k"
server_key: "c0ff1fca0ccb0d855a3e426ae27b7e74"
server_key_iv: "0f07a85d31eb80e6500874ae85ebbaf8"
server_key_version: "encryptionkey@0"
I20260807 11:17:52.551924 13835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:17:52.555477 13850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:52.556839 13835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:17:52.557037 13835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "2032bd73292e4cfa8f42dfc1a5e37b24"
format_stamp: "Formatted at 2026-08-07 11:17:52 on dist-test-slave-5l3k"
server_key: "c0ff1fca0ccb0d855a3e426ae27b7e74"
server_key_iv: "0f07a85d31eb80e6500874ae85ebbaf8"
server_key_version: "encryptionkey@0"
I20260807 11:17:52.557245 13835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:52.594954 13835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:52.599470 13835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:52.599790 13835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:52.619755 13835 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:45845
I20260807 11:17:52.619796 13904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:45845 every 8 connection(s)
I20260807 11:17:52.621165 13835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:17:52.623347 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13835
I20260807 11:17:52.623581 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:17:52.624269 2942 external_mini_cluster.cc:1428] Setting key ead535e026e127af70146840c851545e
I20260807 11:17:52.625313 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:52.637215 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: Bootstrap starting.
Aug 07 11:17:52 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:52.641011 13905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:52.642096 13905 log.cc:824] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:52.645326 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: No bootstrap required, opened a new log
I20260807 11:17:52.646401 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:52.626269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.2.223.190:45845)
0807 11:17:52.627025 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:52.627037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:52.627077 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:17:52.628697 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:52.628718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:52.629088 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:52.629501 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:52.631132 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:52.632489 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:52.633848 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:52.634569 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:52.634785 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:52.639239 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:52.639251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:52.639263 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:52.639305 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:52.642450 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:52.643296 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:52.643304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:52.643312 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:52.643508 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:52.643934 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:52.643937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:52.643939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:52.644418 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:52.644665 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:52.645100 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 11:17:52.645420 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":338,"threads_started":1}
I20260807 11:17:52.652992 13905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } }
I20260807 11:17:52.653311 13905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:52.653445 13905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2032bd73292e4cfa8f42dfc1a5e37b24, State: Initialized, Role: FOLLOWER
I20260807 11:17:52.654034 13905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } }
I20260807 11:17:52.654312 13905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:17:52.654538 13905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:17:52.654745 13905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:52.656864 13905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } }
I20260807 11:17:52.657371 13905 leader_election.cc:302] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2032bd73292e4cfa8f42dfc1a5e37b24; no voters:
I20260807 11:17:52.658155 13905 leader_election.cc:288] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:17:52.658274 13910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:52.658989 13910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [term 1 LEADER]: Becoming Leader. State: Replica: 2032bd73292e4cfa8f42dfc1a5e37b24, State: Running, Role: LEADER
I20260807 11:17:52.659602 13910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } }
I20260807 11:17:52.660495 13905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:17:52.663264 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2032bd73292e4cfa8f42dfc1a5e37b24. Latest consensus state: current_term: 1 leader_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } } }
I20260807 11:17:52.663223 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2032bd73292e4cfa8f42dfc1a5e37b24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45845 } } }
I20260807 11:17:52.663683 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:52.663683 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24 [sys.catalog]: This master's current role is: LEADER
I20260807 11:17:52.665549 13917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:17:52.671239 13917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:17:52.680469 13917 catalog_manager.cc:1347] Generated new cluster ID: b51d7d55900f4b39a8b145bdfcbd2b1a
I20260807 11:17:52.680591 13917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:17:52.699115 13917 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:17:52.701030 13917 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:17:52.732409 13917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: Generated new TSK 0
I20260807 11:17:52.733250 13917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:17:52.815510 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:53.182770 13931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:53.183068 13931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:53.183168 13931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:53.190686 13931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:53.190798 13931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:53.190913 13931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:17:53.201134 13931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:53.202423 13931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:53.203840 13931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:53.218206 13937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:53.218170 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:53.218243 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:53.219564 13931 server_base.cc:1034] running on GCE node
I20260807 11:17:53.220144 13931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:53.221032 13931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:53.222313 13931 hybrid_clock.cc:648] HybridClock initialized: now 1786101473222280 us; error 59 us; skew 500 ppm
I20260807 11:17:53.222702 13931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:53 dist-test-slave-5l3k krb5kdc[13821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101473, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:53.226024 13931 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:17:53.227599 13931 webserver.cc:460] Webserver started at http://127.2.223.129:42281/ using document root <none> and password file <none>
I20260807 11:17:53.228292 13931 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:53.228485 13931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:53.228806 13931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:53.231881 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "4ed3958dd16d448aa343784a7deadc12"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "c70af4d2065ed1bcc56b35b632afa05e"
server_key_iv: "5b60caae5cf86c8a3a91f314e7b564f4"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.232685 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "4ed3958dd16d448aa343784a7deadc12"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "c70af4d2065ed1bcc56b35b632afa05e"
server_key_iv: "5b60caae5cf86c8a3a91f314e7b564f4"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.237587 13931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:17:53.241180 13947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.242615 13931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:53.242794 13931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "4ed3958dd16d448aa343784a7deadc12"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "c70af4d2065ed1bcc56b35b632afa05e"
server_key_iv: "5b60caae5cf86c8a3a91f314e7b564f4"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.243001 13931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:53.291340 13931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:53.296067 13931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:53.296384 13931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:53.297524 13931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:53.299078 13931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:53.299191 13931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.299355 13931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:53.299434 13931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.334417 13931 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:45917
I20260807 11:17:53.334468 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:45917 every 8 connection(s)
I20260807 11:17:53.335786 13931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:17:53.341365 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 13931
I20260807 11:17:53.341610 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:17:53.342381 2942 external_mini_cluster.cc:1428] Setting key ed20def82c74fb96ef411f9c18858a74
Aug 07 11:17:53 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101473, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:53.359349 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:53.338568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:42929 (local address 127.2.223.190:45845)
0807 11:17:53.339069 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:53.339075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:53.340173 (+ 1098us) server_negotiation.cc:407] Connection header received
0807 11:17:53.342134 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:53.342141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:53.342237 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:53.342374 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:53.344042 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.344909 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.347612 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.348046 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.348172 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:53.352743 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:53.352748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:53.352752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:53.352788 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:53.355769 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:53.356628 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:53.356634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:53.356638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:53.356780 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:53.357298 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:53.357309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:53.357312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:53.357614 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:53.357834 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:53.358658 (+ 824us) server_negotiation.cc:299] Negotiation successful
0807 11:17:53.358908 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
I20260807 11:17:53.360545 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:53.339247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45845 (local address 127.2.223.129:42929)
0807 11:17:53.339961 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:53.340014 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:53.341858 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:53.342710 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:53.342718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:53.343284 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:53.343863 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:53.343881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.345105 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.345111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:53.347386 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:53.347407 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.348311 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.348321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:53.348661 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:53.349536 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:53.349581 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:53.352473 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:53.355960 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:53.355970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:53.355997 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:53.356471 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:53.356914 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:53.356920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:53.356924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:53.357152 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:53.357787 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:53.357798 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:53.358194 (+ 396us) client_negotiation.cc:769] Sending connection context
0807 11:17:53.358524 (+ 330us) client_negotiation.cc:240] Negotiation successful
0807 11:17:53.359136 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":285,"threads_started":1}
I20260807 11:17:53.362514 14066 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45845
I20260807 11:17:53.362880 14066 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:53.363808 14066 heartbeater.cc:511] Master 127.2.223.190:45845 requested a full tablet report, sending...
I20260807 11:17:53.366588 13870 ts_manager.cc:194] Registered new tserver with Master: 4ed3958dd16d448aa343784a7deadc12 (127.2.223.129:45917)
I20260807 11:17:53.368906 13870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:42929
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:17:53.415234 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:53.757455 14072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:53.757764 14072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:53.757972 14072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:53.765667 14072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:53.765836 14072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:53.766037 14072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:17:53.776139 14072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:53.777550 14072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:53.779084 14072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:53.792843 14081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:53.792824 14078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:53.793301 14077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:53.793906 14072 server_base.cc:1034] running on GCE node
I20260807 11:17:53.794556 14072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:53.795574 14072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:53.796790 14072 hybrid_clock.cc:648] HybridClock initialized: now 1786101473796721 us; error 60 us; skew 500 ppm
I20260807 11:17:53.797139 14072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:53 dist-test-slave-5l3k krb5kdc[13821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101473, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:53.800458 14072 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:17:53.802004 14072 webserver.cc:460] Webserver started at http://127.2.223.130:37949/ using document root <none> and password file <none>
I20260807 11:17:53.802812 14072 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:53.802950 14072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:53.803298 14072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:53.806288 14072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "5048e40645904af681f72b6ba0a780a1"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "48820881159ad04e8bb48f116bc498be"
server_key_iv: "3ff7a6b8bd89d810952543694acac124"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.807235 14072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "5048e40645904af681f72b6ba0a780a1"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "48820881159ad04e8bb48f116bc498be"
server_key_iv: "3ff7a6b8bd89d810952543694acac124"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.812692 14072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:17:53.816340 14088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.817888 14072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:17:53.818092 14072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "5048e40645904af681f72b6ba0a780a1"
format_stamp: "Formatted at 2026-08-07 11:17:53 on dist-test-slave-5l3k"
server_key: "48820881159ad04e8bb48f116bc498be"
server_key_iv: "3ff7a6b8bd89d810952543694acac124"
server_key_version: "encryptionkey@0"
I20260807 11:17:53.818332 14072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:53.848708 14072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:53.852902 14072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:53.853225 14072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:53.854298 14072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:53.855720 14072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:53.855795 14072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.855880 14072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:53.855932 14072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:53.893337 14072 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:44639
I20260807 11:17:53.893383 14208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:44639 every 8 connection(s)
I20260807 11:17:53.894896 14072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:17:53.904911 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 14072
I20260807 11:17:53.905184 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:17:53.905709 2942 external_mini_cluster.cc:1428] Setting key 62a822ab3fb0fa64a19ea53b41eeb294
Aug 07 11:17:53 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101473, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:53.916666 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:53.898289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45845 (local address 127.2.223.130:57565)
0807 11:17:53.899049 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:53.899096 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:53.900827 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:53.901586 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:53.901595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:53.902106 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:53.902756 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:53.902776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.903842 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.903853 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:53.905130 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:53.905147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.905983 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.905991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:53.906227 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:53.906950 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:53.906985 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:53.909213 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:53.912466 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:53.912475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:53.912492 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:53.912923 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:53.913333 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:53.913338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:53.913341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:53.913539 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:53.914067 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:53.914076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:53.914416 (+ 340us) client_negotiation.cc:769] Sending connection context
0807 11:17:53.914742 (+ 326us) client_negotiation.cc:240] Negotiation successful
0807 11:17:53.915271 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":352,"threads_started":1}
I20260807 11:17:53.917551 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:53.897663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:57565 (local address 127.2.223.190:45845)
0807 11:17:53.898150 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:53.898155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:53.899230 (+ 1075us) server_negotiation.cc:407] Connection header received
0807 11:17:53.901106 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:53.901112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:53.901216 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:53.901342 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:53.902927 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.903683 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.905279 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:53.905735 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:53.905877 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:53.909459 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:53.909465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:53.909468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:53.909499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:53.912294 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:53.913057 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:53.913064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:53.913072 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:53.913209 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:53.913663 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:53.913675 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:53.913678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:53.913929 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:53.914096 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:53.917119 (+ 3023us) server_negotiation.cc:299] Negotiation successful
0807 11:17:53.917289 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260807 11:17:53.918790 14209 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45845
I20260807 11:17:53.919121 14209 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:53.920013 14209 heartbeater.cc:511] Master 127.2.223.190:45845 requested a full tablet report, sending...
I20260807 11:17:53.921680 13870 ts_manager.cc:194] Registered new tserver with Master: 5048e40645904af681f72b6ba0a780a1 (127.2.223.130:44639)
I20260807 11:17:53.922689 13870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:57565
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:17:53.975466 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:17:54.319866 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:17:54.320149 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:17:54.320324 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:17:54.327919 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:17:54.328038 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:17:54.328215 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:17:54.338020 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:17:54.339251 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:17:54.340566 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:17:54.354115 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:54.354053 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:17:54.354053 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:17:54.355125 14215 server_base.cc:1034] running on GCE node
I20260807 11:17:54.355855 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:17:54.356906 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:17:54.358170 14215 hybrid_clock.cc:648] HybridClock initialized: now 1786101474358129 us; error 38 us; skew 500 ppm
I20260807 11:17:54.358598 14215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:17:54 dist-test-slave-5l3k krb5kdc[13821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:17:54.362354 14215 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:17:54.363858 14215 webserver.cc:460] Webserver started at http://127.2.223.131:40415/ using document root <none> and password file <none>
I20260807 11:17:54.364527 14215 fs_manager.cc:359] Metadata directory not provided
I20260807 11:17:54.364639 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:17:54.364916 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:17:54.368070 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "b769593f14424dce8077dc3727bc10c2"
format_stamp: "Formatted at 2026-08-07 11:17:54 on dist-test-slave-5l3k"
server_key: "750319affaf29ab049cca18d0ce0ab73"
server_key_iv: "a8d6c093874ecec4af062d47b65b49a1"
server_key_version: "encryptionkey@0"
I20260807 11:17:54.368836 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "b769593f14424dce8077dc3727bc10c2"
format_stamp: "Formatted at 2026-08-07 11:17:54 on dist-test-slave-5l3k"
server_key: "750319affaf29ab049cca18d0ce0ab73"
server_key_iv: "a8d6c093874ecec4af062d47b65b49a1"
server_key_version: "encryptionkey@0"
I20260807 11:17:54.373144 14066 heartbeater.cc:503] Master 127.2.223.190:45845 was elected leader, sending a full tablet report...
I20260807 11:17:54.374452 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:17:54.378352 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:54.379721 14215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:17:54.379899 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "b769593f14424dce8077dc3727bc10c2"
format_stamp: "Formatted at 2026-08-07 11:17:54 on dist-test-slave-5l3k"
server_key: "750319affaf29ab049cca18d0ce0ab73"
server_key_iv: "a8d6c093874ecec4af062d47b65b49a1"
server_key_version: "encryptionkey@0"
I20260807 11:17:54.380095 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:17:54.391562 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:17:54.396070 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:17:54.396392 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:17:54.397457 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:17:54.398947 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:17:54.399022 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:54.399180 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:17:54.399266 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:17:54.434897 14215 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:40139
I20260807 11:17:54.434944 14349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:40139 every 8 connection(s)
I20260807 11:17:54.436251 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:17:54.437326 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 14215
I20260807 11:17:54.437548 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:17:54.438115 2942 external_mini_cluster.cc:1428] Setting key 5f293385d0d8b09a63e68ba726ca8159
Aug 07 11:17:54 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:17:54.458179 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.439743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:33213 (local address 127.2.223.190:45845)
0807 11:17:54.440350 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.440359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.441171 (+ 812us) server_negotiation.cc:407] Connection header received
0807 11:17:54.443254 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.443259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.443327 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.443443 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:54.445070 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.445814 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.447912 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.448344 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.448451 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.452007 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:54.452012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:54.452016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:54.452049 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:54.455271 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.456023 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.456027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.456029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.456177 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.456577 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.456581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.456585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.456858 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:54.456971 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.457684 (+ 713us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.457871 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":295,"threads_started":1}
I20260807 11:17:54.459252 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.439913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45845 (local address 127.2.223.131:33213)
0807 11:17:54.440962 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.441021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.442958 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.443693 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.443702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.444317 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:17:54.444904 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.444922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.445955 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.445959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.447773 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.447790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.448588 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.448594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.448791 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.449469 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:17:54.449492 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:17:54.451790 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:17:54.455433 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:54.455440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:54.455454 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:54.455904 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:54.456282 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:17:54.456285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:17:54.456287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:17:54.456465 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:17:54.457000 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:17:54.457005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:17:54.457327 (+ 322us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.457574 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.458001 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":488,"threads_started":1}
I20260807 11:17:54.460767 14350 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45845
I20260807 11:17:54.461135 14350 heartbeater.cc:464] Registering TS with master...
I20260807 11:17:54.461917 14350 heartbeater.cc:511] Master 127.2.223.190:45845 requested a full tablet report, sending...
I20260807 11:17:54.463567 13870 ts_manager.cc:194] Registered new tserver with Master: b769593f14424dce8077dc3727bc10c2 (127.2.223.131:40139)
I20260807 11:17:54.464455 13870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:33213
I20260807 11:17:54.466979 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:17:54.485438 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.471975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.2.223.190:45845)
0807 11:17:54.472190 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.472194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.472749 (+ 555us) server_negotiation.cc:407] Connection header received
0807 11:17:54.472892 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.472895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.472975 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.473061 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:54.474553 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.475405 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.476654 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.477095 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.477204 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.479603 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:54.479609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:54.479612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:54.479647 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:54.482455 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.483270 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.483274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.483277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.483453 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.483923 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.483928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.483932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.484305 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:54.484563 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.484952 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.485178 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:17:54.489993 13870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:17:54.497001 13870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:17:54.529637 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.509849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:40139 (local address 127.0.0.1:37908)
0807 11:17:54.518782 (+ 8933us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.518824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.519028 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.520991 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.521004 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.521029 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:54.521485 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.521497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524272 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.524278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.526229 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.526242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.528351 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.528357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.528569 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.528623 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.528827 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.529099 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8670,"mutex_wait_us":1,"thread_start_us":11808,"threads_started":1}
I20260807 11:17:54.529723 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.510660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.2.223.131:40139)
0807 11:17:54.519932 (+ 9272us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.519944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.519984 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:17:54.520107 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.520117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.520409 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.520635 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:54.521696 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524117 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.526405 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.527856 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.527978 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.528226 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.528865 (+ 639us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.529326 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9084,"thread_start_us":8910,"threads_started":1}
I20260807 11:17:54.530830 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.510592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:44639 (local address 127.0.0.1:49838)
0807 11:17:54.521581 (+ 10989us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.521600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.521814 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.522760 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.522764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.522791 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:54.523231 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.523245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524749 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.524758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.527193 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.527201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.529531 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.529540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.529990 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.530036 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.530192 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.530375 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10865,"mutex_wait_us":67,"thread_start_us":7750,"threads_started":1}
I20260807 11:17:54.530846 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.511285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.2.223.130:44639)
0807 11:17:54.520230 (+ 8945us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.520238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.521699 (+ 1461us) server_negotiation.cc:407] Connection header received
0807 11:17:54.522018 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.522026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.522219 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.522439 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:54.523423 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524600 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.527341 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.528967 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.529103 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.529354 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.530278 (+ 924us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.530498 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8790,"thread_start_us":8653,"threads_started":1}
I20260807 11:17:54.530978 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.509920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45917 (local address 127.0.0.1:34888)
0807 11:17:54.520678 (+ 10758us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.520698 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.520816 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.521920 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.521924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.521952 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:54.522320 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.522354 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524314 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.524318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.527164 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.527185 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.529800 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.529809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.530284 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.530324 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.530423 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.530568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10626,"mutex_wait_us":140,"thread_start_us":10476,"threads_started":1}
I20260807 11:17:54.531688 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.510016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.2.223.129:45917)
0807 11:17:54.521186 (+ 11170us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.521197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.521228 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:17:54.521363 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.521372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.521634 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.521839 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:54.522583 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.524117 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.527383 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.528904 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.529079 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.529556 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.531104 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.531295 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10972,"thread_start_us":10789,"threads_started":1}
I20260807 11:17:54.534278 14281 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd41e4d3851476bb40755d974ff3113 (DEFAULT_TABLE table=test-table [id=0e8611e8781a44d1912cbb27f07b74f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:54.534699 14139 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd41e4d3851476bb40755d974ff3113 (DEFAULT_TABLE table=test-table [id=0e8611e8781a44d1912cbb27f07b74f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:54.535578 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd41e4d3851476bb40755d974ff3113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:54.535236 13997 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd41e4d3851476bb40755d974ff3113 (DEFAULT_TABLE table=test-table [id=0e8611e8781a44d1912cbb27f07b74f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:17:54.536024 14139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd41e4d3851476bb40755d974ff3113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:54.536499 13997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd41e4d3851476bb40755d974ff3113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:17:54.546943 14372 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Bootstrap starting.
I20260807 11:17:54.551234 14373 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Bootstrap starting.
I20260807 11:17:54.551610 14372 tablet_bootstrap.cc:654] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:54.552917 14374 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Bootstrap starting.
I20260807 11:17:54.553522 14372 log.cc:824] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:54.554999 14373 tablet_bootstrap.cc:654] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:54.556221 14374 tablet_bootstrap.cc:654] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Neither blocks nor log segments found. Creating new log.
I20260807 11:17:54.556344 14373 log.cc:824] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:54.557129 14372 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: No bootstrap required, opened a new log
I20260807 11:17:54.557417 14372 ts_tablet_manager.cc:1397] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260807 11:17:54.557528 14374 log.cc:824] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Log is configured to *not* fsync() on all Append() calls
I20260807 11:17:54.559597 14373 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: No bootstrap required, opened a new log
I20260807 11:17:54.559872 14373 ts_tablet_manager.cc:1397] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 11:17:54.560846 14374 tablet_bootstrap.cc:492] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: No bootstrap required, opened a new log
I20260807 11:17:54.561101 14374 ts_tablet_manager.cc:1397] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 11:17:54.566704 14372 raft_consensus.cc:348] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.567180 14372 raft_consensus.cc:374] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:54.567286 14372 raft_consensus.cc:729] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b769593f14424dce8077dc3727bc10c2, State: Initialized, Role: FOLLOWER
I20260807 11:17:54.567940 14372 consensus_queue.cc:260] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.570811 14373 raft_consensus.cc:348] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.571291 14373 raft_consensus.cc:374] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:54.571406 14373 raft_consensus.cc:729] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5048e40645904af681f72b6ba0a780a1, State: Initialized, Role: FOLLOWER
I20260807 11:17:54.571868 14374 raft_consensus.cc:348] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.572026 14373 consensus_queue.cc:260] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.572290 14374 raft_consensus.cc:374] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:17:54.572395 14374 raft_consensus.cc:729] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed3958dd16d448aa343784a7deadc12, State: Initialized, Role: FOLLOWER
I20260807 11:17:54.573117 14374 consensus_queue.cc:260] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.575186 14350 heartbeater.cc:503] Master 127.2.223.190:45845 was elected leader, sending a full tablet report...
I20260807 11:17:54.576268 14372 ts_tablet_manager.cc:1428] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260807 11:17:54.577782 14374 ts_tablet_manager.cc:1428] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20260807 11:17:54.578872 14209 heartbeater.cc:503] Master 127.2.223.190:45845 was elected leader, sending a full tablet report...
I20260807 11:17:54.579427 14373 ts_tablet_manager.cc:1428] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Time spent starting tablet: real 0.019s user 0.015s sys 0.005s
W20260807 11:17:54.588274 14067 tablet.cc:2369] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:54.647343 14210 tablet.cc:2369] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:17:54.688186 14351 tablet.cc:2369] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:17:54.688251 14379 raft_consensus.cc:482] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:17:54.688604 14379 raft_consensus.cc:504] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.690323 14379 leader_election.cc:288] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed3958dd16d448aa343784a7deadc12 (127.2.223.129:45917), b769593f14424dce8077dc3727bc10c2 (127.2.223.131:40139)
I20260807 11:17:54.700433 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.690811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:46967 (local address 127.2.223.131:40139)
0807 11:17:54.691051 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.691060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.691893 (+ 833us) server_negotiation.cc:407] Connection header received
0807 11:17:54.692008 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.692013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.692105 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.692233 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:54.693104 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.694410 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.697593 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.699019 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.699208 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.699291 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.699989 (+ 698us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.700138 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:17:54.700436 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.690982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:42799 (local address 127.2.223.129:45917)
0807 11:17:54.691247 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.691256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.691800 (+ 544us) server_negotiation.cc:407] Connection header received
0807 11:17:54.692020 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.692025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.692143 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.692247 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:17:54.693180 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.694421 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.697580 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.699019 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.699208 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.699291 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.699989 (+ 698us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.700165 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 11:17:54.700649 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.690737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:40139 (local address 127.2.223.130:46967)
0807 11:17:54.691777 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.691794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.691893 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.692281 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.692287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.692380 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:54.692881 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.692899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.694586 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.694591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.697389 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.697407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.699338 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.699345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.699702 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.699829 (+ 127us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.699933 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.700110 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":402,"spinlock_wait_cycles":24320,"thread_start_us":392,"threads_started":1}
I20260807 11:17:54.701097 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.690737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45917 (local address 127.2.223.130:42799)
0807 11:17:54.691638 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0807 11:17:54.691666 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:17:54.691854 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:17:54.692281 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:17:54.692287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:17:54.692316 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:17:54.692983 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.692996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.694585 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.694591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.697399 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:17:54.697407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.699340 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.699345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:17:54.699706 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.699834 (+ 128us) client_negotiation.cc:769] Sending connection context
0807 11:17:54.699933 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:17:54.700109 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":335,"thread_start_us":699,"threads_started":1}
I20260807 11:17:54.701938 14302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd41e4d3851476bb40755d974ff3113" candidate_uuid: "5048e40645904af681f72b6ba0a780a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b769593f14424dce8077dc3727bc10c2" is_pre_election: true
I20260807 11:17:54.702611 14302 raft_consensus.cc:2454] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5048e40645904af681f72b6ba0a780a1 in term 0.
I20260807 11:17:54.702597 14019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd41e4d3851476bb40755d974ff3113" candidate_uuid: "5048e40645904af681f72b6ba0a780a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed3958dd16d448aa343784a7deadc12" is_pre_election: true
I20260807 11:17:54.703133 14019 raft_consensus.cc:2454] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5048e40645904af681f72b6ba0a780a1 in term 0.
I20260807 11:17:54.703320 14090 leader_election.cc:302] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5048e40645904af681f72b6ba0a780a1, b769593f14424dce8077dc3727bc10c2; no voters:
I20260807 11:17:54.703794 14379 raft_consensus.cc:2781] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:17:54.703958 14379 raft_consensus.cc:482] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:17:54.704075 14379 raft_consensus.cc:3037] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:54.706267 14379 raft_consensus.cc:504] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.706979 14379 leader_election.cc:288] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4ed3958dd16d448aa343784a7deadc12 (127.2.223.129:45917), b769593f14424dce8077dc3727bc10c2 (127.2.223.131:40139)
I20260807 11:17:54.707409 14019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd41e4d3851476bb40755d974ff3113" candidate_uuid: "5048e40645904af681f72b6ba0a780a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed3958dd16d448aa343784a7deadc12"
I20260807 11:17:54.707489 14302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd41e4d3851476bb40755d974ff3113" candidate_uuid: "5048e40645904af681f72b6ba0a780a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b769593f14424dce8077dc3727bc10c2"
I20260807 11:17:54.707652 14019 raft_consensus.cc:3037] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:54.707734 14302 raft_consensus.cc:3037] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:17:54.709910 14019 raft_consensus.cc:2454] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5048e40645904af681f72b6ba0a780a1 in term 1.
I20260807 11:17:54.709942 14302 raft_consensus.cc:2454] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5048e40645904af681f72b6ba0a780a1 in term 1.
I20260807 11:17:54.710355 14092 leader_election.cc:302] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed3958dd16d448aa343784a7deadc12, 5048e40645904af681f72b6ba0a780a1; no voters:
I20260807 11:17:54.710690 14379 raft_consensus.cc:2781] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:17:54.711261 14379 raft_consensus.cc:686] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 1 LEADER]: Becoming Leader. State: Replica: 5048e40645904af681f72b6ba0a780a1, State: Running, Role: LEADER
I20260807 11:17:54.711822 14379 consensus_queue.cc:237] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } }
I20260807 11:17:54.716514 13869 catalog_manager.cc:5495] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5048e40645904af681f72b6ba0a780a1 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "5048e40645904af681f72b6ba0a780a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5048e40645904af681f72b6ba0a780a1" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 44639 } health_report { overall_health: HEALTHY } } }
I20260807 11:17:54.761214 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.755696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.2.223.130:44639)
0807 11:17:54.755964 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.755968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.755989 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:17:54.756089 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.756093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.756181 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.756310 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:17:54.756658 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.757384 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.758831 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.759263 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.759367 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.759816 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:17:54.760473 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:17:54.760635 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.760742 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.760809 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 11:17:54.769958 14302 raft_consensus.cc:1261] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 1 FOLLOWER]: Refusing update from remote peer 5048e40645904af681f72b6ba0a780a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:54.770033 14019 raft_consensus.cc:1261] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 1 FOLLOWER]: Refusing update from remote peer 5048e40645904af681f72b6ba0a780a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:17:54.771063 14383 consensus_queue.cc:1035] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed3958dd16d448aa343784a7deadc12" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:54.771432 14379 consensus_queue.cc:1035] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b769593f14424dce8077dc3727bc10c2" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 40139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:17:54.787065 14387 mvcc.cc:205] Tried to move back new op lower bound from 7315871640647065600 to 7315871640423755776. Current Snapshot: MvccSnapshot[applied={T|T < 7315871640647065600 or (T in {7315871640647065600})}]
I20260807 11:17:54.800452 13869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:54.800740 13869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:17:54.804526 13869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2032bd73292e4cfa8f42dfc1a5e37b24: Sending DeleteTablet for 3 replicas of tablet 4dd41e4d3851476bb40755d974ff3113
I20260807 11:17:54.805711 13997 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd41e4d3851476bb40755d974ff3113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:54 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:34888
I20260807 11:17:54.805701 14139 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd41e4d3851476bb40755d974ff3113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:54 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:49838
I20260807 11:17:54.805692 14281 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd41e4d3851476bb40755d974ff3113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:17:54 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37908
I20260807 11:17:54.812507 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13931
I20260807 11:17:54.819280 14398 tablet_replica.cc:330] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: stopping tablet replica
I20260807 11:17:54.819911 14398 raft_consensus.cc:2229] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:17:54.820809 14398 raft_consensus.cc:2258] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:54.821839 14396 tablet_replica.cc:330] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: stopping tablet replica
I20260807 11:17:54.822443 14397 tablet_replica.cc:330] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: stopping tablet replica
I20260807 11:17:54.823480 14396 raft_consensus.cc:2229] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:54.823801 14398 ts_tablet_manager.cc:1905] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:54.824028 14396 raft_consensus.cc:2258] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:54.822896 14397 raft_consensus.cc:2229] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:17:54.824766 14397 raft_consensus.cc:2258] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:17:54.826752 14397 ts_tablet_manager.cc:1905] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:17:54 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:17:54.832064 14398 ts_tablet_manager.cc:1918] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:54.832248 14398 log.cc:1196] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/4dd41e4d3851476bb40755d974ff3113
I20260807 11:17:54.832494 14396 ts_tablet_manager.cc:1905] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:17:54.832914 14398 ts_tablet_manager.cc:1939] T 4dd41e4d3851476bb40755d974ff3113 P 5048e40645904af681f72b6ba0a780a1: Deleting consensus metadata
I20260807 11:17:54.833802 14397 ts_tablet_manager.cc:1918] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:54.833966 14397 log.cc:1196] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/4dd41e4d3851476bb40755d974ff3113
I20260807 11:17:54.834607 14397 ts_tablet_manager.cc:1939] T 4dd41e4d3851476bb40755d974ff3113 P b769593f14424dce8077dc3727bc10c2: Deleting consensus metadata
I20260807 11:17:54.834808 13854 catalog_manager.cc:4841] TS 5048e40645904af681f72b6ba0a780a1 (127.2.223.130:44639): tablet 4dd41e4d3851476bb40755d974ff3113 (table test-table [id=0e8611e8781a44d1912cbb27f07b74f7]) successfully deleted
I20260807 11:17:54.836364 13853 catalog_manager.cc:4841] TS b769593f14424dce8077dc3727bc10c2 (127.2.223.131:40139): tablet 4dd41e4d3851476bb40755d974ff3113 (table test-table [id=0e8611e8781a44d1912cbb27f07b74f7]) successfully deleted
I20260807 11:17:54.840956 14396 ts_tablet_manager.cc:1918] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:17:54.841151 14396 log.cc:1196] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/4dd41e4d3851476bb40755d974ff3113
I20260807 11:17:54.841749 14396 ts_tablet_manager.cc:1939] T 4dd41e4d3851476bb40755d974ff3113 P 4ed3958dd16d448aa343784a7deadc12: Deleting consensus metadata
I20260807 11:17:54.842674 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.814226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.2.223.129:45917)
0807 11:17:54.814429 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.814435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.814456 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:17:54.814918 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.814924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.815014 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.815127 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:54.822235 (+ 7108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.823455 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.825572 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.826190 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.826290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.835727 (+ 9437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:54.835743 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:54.835760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:54.835806 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:54.839455 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.840249 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.840257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.840259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.840409 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.840830 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.840840 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.840842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.841352 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:54.841707 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.841963 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.842201 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:17:54.843412 14059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:54.843629 13855 catalog_manager.cc:4841] TS 4ed3958dd16d448aa343784a7deadc12 (127.2.223.129:45917): tablet 4dd41e4d3851476bb40755d974ff3113 (table test-table [id=0e8611e8781a44d1912cbb27f07b74f7]) successfully deleted
I20260807 11:17:54.956043 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13931
I20260807 11:17:54.978107 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 14072
Aug 07 11:17:54 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:17:54.995286 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:54.979270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.2.223.130:44639)
0807 11:17:54.979430 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:54.979435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:54.979449 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:17:54.979502 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:54.979506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:54.979591 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:54.979723 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:54.980954 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.981774 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.983156 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:54.983635 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:54.983795 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:54.988555 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:54.988571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:54.988587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:54.988630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:54.992069 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.992923 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.992932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.992934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.993103 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:54.993549 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:54.993553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:54.993555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:54.993942 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:54.994211 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:54.994670 (+ 459us) server_negotiation.cc:299] Negotiation successful
0807 11:17:54.994848 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 11:17:54.996147 14204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:55.115206 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 14072
I20260807 11:17:55.138377 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 14215
Aug 07 11:17:55 dist-test-slave-5l3k krb5kdc[13821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:17:55.153671 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:17:55.139291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.2.223.131:40139)
0807 11:17:55.139475 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 11:17:55.139480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:17:55.139499 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:17:55.139677 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:17:55.139681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:17:55.139758 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:17:55.139838 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:17:55.141239 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:55.141916 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:55.143242 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:17:55.143610 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:17:55.143700 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:17:55.148067 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:17:55.148086 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:17:55.148104 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:17:55.148143 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:17:55.151203 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:55.151828 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:55.151835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:55.151837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:55.151981 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:17:55.152314 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:17:55.152317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:17:55.152319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:17:55.152678 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:17:55.152899 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 11:17:55.153146 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 11:17:55.153296 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:17:55.154332 14345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:55.245134 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 14215
I20260807 11:17:55.266562 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 13835
I20260807 11:17:55.267762 13900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:17:55.382058 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 13835
2026-08-07T11:17:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4718 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:17:55 dist-test-slave-5l3k krb5kdc[14408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:17:55 dist-test-slave-5l3k krb5kdc[14408](info): set up 2 sockets
Aug 07 11:17:55 dist-test-slave-5l3k krb5kdc[14408](info): commencing operation
krb5kdc: starting...
W20260807 11:17:56.547956 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.008s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:17:56 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:17:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangeradmin_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangeradmin_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangerlookup_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangerlookup_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
I20260807 11:17:56.970664 2942 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgCuO6z/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/postgres -l logfile start
2026-08-07 11:17:59.637 UTC [14441] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 11:17:59.637 UTC [14441] LOG: listening on IPv4 address "127.2.223.148", port 36331
2026-08-07 11:17:59.641 UTC [14441] LOG: listening on Unix socket "/tmp/.s.PGSQL.36331"
2026-08-07 11:17:59.665 UTC [14444] LOG: database system was shut down at 2026-08-07 11:17:58 UTC
2026-08-07 11:17:59.670 UTC [14441] LOG: database system is ready to accept connections
I20260807 11:18:00.684968 2942 mini_postgres.cc:96] Postgres bound to 36331
2026-08-07 11:18:00.711 UTC [14454] FATAL: database "slave" does not exist
127.2.223.148:36331 - accepting connections
I20260807 11:18:00.712257 2942 mini_ranger.cc:161] Starting Ranger...
I20260807 11:18:00.745723 2942 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 11:18:01.286787 2942 mini_ranger.cc:87] Created ranger Postgres database
I20260807 11:18:01.286904 2942 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin
2026-08-07 11:18:01,590 [I] DB FLAVOR :POSTGRES
2026-08-07 11:18:01,591 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:18:01,591 [I] Checking connection..
2026-08-07 11:18:01,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 11:18:01,857 [I] Checking connection passed.
2026-08-07 11:18:01,858 [I] --------- Verifying version history table ---------
2026-08-07 11:18:01,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:02,124 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:18:02,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:02,401 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:18:02,401 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 11:18:02,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 11:18:02.648 UTC [14522] WARNING: there is no transaction in progress
2026-08-07 11:18:02,655 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 11:18:02,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:02,921 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 11:18:02,922 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:18:02,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 11:18:03,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 11:18:03,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5l3k.c.gcp-upstream.internal','N') ;"
2026-08-07 11:18:03,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:18:03,931 [I] Table x_portal_user does not exist in database ranger
2026-08-07 11:18:03,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:18:04,195 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 11:18:04,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:18:04,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:18:04,697 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 11:18:04,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 11:18:05.656 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.673 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.690 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.705 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.721 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.740 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.796 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.803 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.817 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.827 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.838 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.852 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.862 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.869 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.875 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:05.882 UTC [14633] WARNING: there is no transaction in progress
2026-08-07 11:18:06,529 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 11:18:06,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 11:18:06,767 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 11:18:06,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:18:07,013 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 11:18:07,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:18:07,271 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 11:18:07,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:18:07,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:18:07,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5l3k.c.gcp-upstream.internal';"
2026-08-07 11:18:08,037 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 11:18:08,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 11:18:08,296 [I] DB_PATCHES have already been applied
I20260807 11:18:08.302361 2942 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgCuO6z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 11:18:08.302451 2942 mini_ranger.cc:193] Using host: 127.2.223.148
I20260807 11:18:08.325523 2942 mini_ranger.cc:227] Ranger admin URL: http://127.2.223.148:38321
Aug 07, 2026 11:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 11:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 11:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 11:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 11:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 11:18:09.523622 2942 mini_ranger.cc:160] Time spent starting Ranger: real 8.811s user 0.003s sys 0.063s
Aug 07, 2026 11:18:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 11:18:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 07, 2026 11:18:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 11:18:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 11:18:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 11:18:31.865680 2942 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangerkms_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/rangerkms_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.223.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.223.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 11:18:32 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101512, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 11:18:32.090675 2942 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 11:18:32.122295 2942 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 11:18:32.977488 2942 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 11:18:32.977689 2942 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms
I20260807 11:18:32.977854 2942 mini_ranger_kms.cc:227] Using postgres at 127.2.223.148:36331
2026-08-07 11:18:33,493 [I] DB FLAVOR :POSTGRES
2026-08-07 11:18:33,494 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:18:33,494 [I] Checking connection
2026-08-07 11:18:33,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 11:18:33,757 [I] connection success
2026-08-07 11:18:33,758 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 11:18:33,758 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 11:18:33,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 11:18:33,994 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 11:18:33,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:18:33,995 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 11:18:33,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:36331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 11:18:34,262 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 11:18:34.711555 2942 mini_ranger_kms.cc:310] Created kms service
I20260807 11:18:34.932865 2942 mini_ranger_kms.cc:326] Created kudu user
I20260807 11:18:34.985546 2942 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 11:18:34 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101512, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:35.416800 2942 mini_ranger_kms.cc:384] Added ranger policy
I20260807 11:18:35.418123 2942 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 11:18:35.418215 2942 mini_ranger_kms.cc:240] Using host: 127.2.223.148
I20260807 11:18:35.447851 2942 mini_ranger_kms.cc:276] Ranger KMS PID: 14826
I20260807 11:18:35.448060 2942 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.223.148:38039
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 11:18:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 11:18:37 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786101517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:18:37.863376 2942 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.773s user 0.151s sys 0.418s
I20260807 11:18:37.863845 2942 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 11:18:37.864071 2942 mini_ranger_kms.cc:401] 127.2.223.148:38039/kms/v1/keys
Aug 07 11:18:38 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
Aug 07 11:18:38 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
Aug 07 11:18:41 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:18:41.206362 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:38653
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:38653
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:38039/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:18:41.793883 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:18:41.794221 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:18:41.794414 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:18:41.802450 14876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:18:41.802625 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:18:41.802747 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:18:41.802883 14876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:18:41.802982 14876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:18:41.814257 14876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.223.148:38039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:38653
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:18:41.815934 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:18:41.817265 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:18:41.832605 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:41.832981 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:41.833209 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:18:41.834667 14876 server_base.cc:1034] running on GCE node
I20260807 11:18:41.835822 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:18:41.837711 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:18:41.839042 14876 hybrid_clock.cc:648] HybridClock initialized: now 1786101521838927 us; error 57 us; skew 500 ppm
I20260807 11:18:41.839565 14876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:18:41 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:18:41.844010 14876 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:18:41.846405 14876 webserver.cc:460] Webserver started at http://127.2.223.190:44197/ using document root <none> and password file <none>
I20260807 11:18:41.847317 14876 fs_manager.cc:359] Metadata directory not provided
I20260807 11:18:41.847558 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:18:41.847934 14876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:18:41 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:41.912420 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64"
format_stamp: "Formatted at 2026-08-07 11:18:41 on dist-test-slave-5l3k"
server_key: "89cf39f6e61abe14a0cabdfb797d3d11"
server_key_iv: "aeec1dc6a4af1de31ead6ef1a7812ec2"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:41.913910 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64"
format_stamp: "Formatted at 2026-08-07 11:18:41 on dist-test-slave-5l3k"
server_key: "89cf39f6e61abe14a0cabdfb797d3d11"
server_key_iv: "aeec1dc6a4af1de31ead6ef1a7812ec2"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:41.923417 14876 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.003s
I20260807 11:18:41.945787 14892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:41.947652 14876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:18:41.947907 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64"
format_stamp: "Formatted at 2026-08-07 11:18:41 on dist-test-slave-5l3k"
server_key: "89cf39f6e61abe14a0cabdfb797d3d11"
server_key_iv: "aeec1dc6a4af1de31ead6ef1a7812ec2"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:41.948283 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:18:41.980190 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:18:41.985888 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:18:41.986436 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:18:42.033761 14876 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:38653
I20260807 11:18:42.033983 14948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:38653 every 8 connection(s)
I20260807 11:18:42.035372 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:18:42.039345 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 14876
I20260807 11:18:42.039572 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:18:42.040314 2942 external_mini_cluster.cc:1428] Setting key a3e513dccc30943e8ae097d15357173b
I20260807 11:18:42.041538 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:18:42 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:18:42.061837 14950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: Bootstrap starting.
I20260807 11:18:42.068686 14950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: Neither blocks nor log segments found. Creating new log.
I20260807 11:18:42.071257 14950 log.cc:824] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: Log is configured to *not* fsync() on all Append() calls
I20260807 11:18:42.075050 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:42.042439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.2.223.190:38653)
0807 11:18:42.044131 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:42.044149 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:42.044252 (+ 103us) server_negotiation.cc:407] Connection header received
0807 11:18:42.047082 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:42.047111 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:42.047804 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:42.048411 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:42.049581 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:42.052942 (+ 3361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:42.054520 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:42.055547 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:42.055784 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:42.060614 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:42.060624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:42.060639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:42.060678 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:42.067269 (+ 6591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:42.068172 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:42.068181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:42.068190 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:42.068446 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:42.069089 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:42.069094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:42.069097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:42.070457 (+ 1360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:42.071005 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:42.072126 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0807 11:18:42.072721 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":977,"threads_started":1}
I20260807 11:18:42.078250 14950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: No bootstrap required, opened a new log
I20260807 11:18:42.090250 14950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } }
I20260807 11:18:42.090845 14950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:18:42.091045 14950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b2bde6a8b146158c9e2c5e8e9b9e64, State: Initialized, Role: FOLLOWER
I20260807 11:18:42.091722 14950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } }
I20260807 11:18:42.092105 14950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:18:42.092339 14950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:18:42.092584 14950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:18:42.095765 14950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } }
I20260807 11:18:42.096413 14950 leader_election.cc:302] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b2bde6a8b146158c9e2c5e8e9b9e64; no voters:
I20260807 11:18:42.097451 14950 leader_election.cc:288] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:18:42.097666 14955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:18:42.099462 14955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [term 1 LEADER]: Becoming Leader. State: Replica: 48b2bde6a8b146158c9e2c5e8e9b9e64, State: Running, Role: LEADER
I20260807 11:18:42.100265 14955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } }
I20260807 11:18:42.100920 14950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:18:42.104543 14957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b2bde6a8b146158c9e2c5e8e9b9e64. Latest consensus state: current_term: 1 leader_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } } }
I20260807 11:18:42.105764 14957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [sys.catalog]: This master's current role is: LEADER
I20260807 11:18:42.105262 14956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2bde6a8b146158c9e2c5e8e9b9e64" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 38653 } } }
I20260807 11:18:42.106227 14956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64 [sys.catalog]: This master's current role is: LEADER
I20260807 11:18:42.111924 14962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:18:42.119834 14962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:18:42.120754 14950 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 11:18:42.146818 14962 catalog_manager.cc:1347] Generated new cluster ID: 3121940a131440e69ebaba6f34d680b8
I20260807 11:18:42.147049 14962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:18:42.184427 14962 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:18:42.186096 14962 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:18:42.195506 14962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: Generated new TSK 0
I20260807 11:18:42.196987 14962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:18:43.060882 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:38039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:18:43 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 11:18:43.780480 14997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:18:43.780977 14997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:18:43.781270 14997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:18:43.795128 14997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:18:43.795416 14997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:18:43.795627 14997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:18:43.817132 14997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.223.148:38039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:18:43.819542 14997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:18:43.821643 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:18:43.844676 15008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:43.845357 15007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:43.847875 15010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:18:43.849288 14997 server_base.cc:1034] running on GCE node
I20260807 11:18:43.850442 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:18:43.852214 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:18:43.853653 14997 hybrid_clock.cc:648] HybridClock initialized: now 1786101523853487 us; error 71 us; skew 500 ppm
I20260807 11:18:43.854310 14997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:18:43 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:18:43.861752 14997 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:18:43.865265 14997 webserver.cc:460] Webserver started at http://127.2.223.129:43515/ using document root <none> and password file <none>
I20260807 11:18:43.867065 14997 fs_manager.cc:359] Metadata directory not provided
I20260807 11:18:43.867352 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:18:43.868194 14997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:18:43 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:43.893926 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "a4fea3f1510e49939a41a0fb34d14c54"
format_stamp: "Formatted at 2026-08-07 11:18:43 on dist-test-slave-5l3k"
server_key: "d9d2cee9160e71b5f4626ec608d216bf"
server_key_iv: "4b48e135ecbc46b0d72df17194ae00a6"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:43.895126 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "a4fea3f1510e49939a41a0fb34d14c54"
format_stamp: "Formatted at 2026-08-07 11:18:43 on dist-test-slave-5l3k"
server_key: "d9d2cee9160e71b5f4626ec608d216bf"
server_key_iv: "4b48e135ecbc46b0d72df17194ae00a6"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:43.917941 14997 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.008s sys 0.005s
I20260807 11:18:43.945765 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:43.949057 14997 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s
I20260807 11:18:43.949405 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "a4fea3f1510e49939a41a0fb34d14c54"
format_stamp: "Formatted at 2026-08-07 11:18:43 on dist-test-slave-5l3k"
server_key: "d9d2cee9160e71b5f4626ec608d216bf"
server_key_iv: "4b48e135ecbc46b0d72df17194ae00a6"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:43.949777 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:18:44.032634 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:18:44.040200 14997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:18:44.040830 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:18:44.054543 14997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:18:44.079398 14997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:18:44.079669 14997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:44.080061 14997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:18:44.080226 14997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:18:44 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:44.502928 14997 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:35445
I20260807 11:18:44.503052 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:35445 every 8 connection(s)
I20260807 11:18:44.505115 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:18:44.514623 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 14997
I20260807 11:18:44.514873 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:18:44.515985 2942 external_mini_cluster.cc:1428] Setting key f3f8e4c33c245b9fde4844ec22f83c95
Aug 07 11:18:44 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:18:44.554247 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:44.509964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:38653 (local address 127.2.223.129:44521)
0807 11:18:44.517638 (+ 7674us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:44.517719 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:44.521463 (+ 3744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:44.528733 (+ 7270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:44.528747 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:44.529759 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:18:44.530919 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:44.531012 (+ 93us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:44.532803 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:44.532812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:44.535752 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:44.535770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:44.537215 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:44.537225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:44.537644 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:44.538844 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:18:44.538883 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:18:44.543012 (+ 4129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:18:44.548532 (+ 5520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:44.548543 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:44.548561 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:44.549220 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:44.549851 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:44.549856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:44.549859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:44.550350 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:44.551293 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:18:44.551301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:18:44.551757 (+ 456us) client_negotiation.cc:769] Sending connection context
0807 11:18:44.552061 (+ 304us) client_negotiation.cc:240] Negotiation successful
0807 11:18:44.552707 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7190,"thread_start_us":3774,"threads_started":1}
I20260807 11:18:44.557901 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:44.509106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:44521 (local address 127.2.223.190:38653)
0807 11:18:44.525766 (+ 16660us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:44.525772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:44.525796 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:18:44.525860 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:44.525865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:44.525949 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:44.526077 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:44.531217 (+ 5140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:44.532547 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:44.536011 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:44.536899 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:44.541872 (+ 4973us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:44.543339 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:44.543346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:44.543351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:44.543389 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:44.548354 (+ 4965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:44.549389 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:44.549395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:44.549399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:44.549663 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:44.550579 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:44.550585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:44.550588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:44.551125 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:44.552611 (+ 1486us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:44.552771 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 11:18:44.553019 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16410,"thread_start_us":17059,"threads_started":1}
I20260807 11:18:44.560652 15137 heartbeater.cc:347] Connected to a master server at 127.2.223.190:38653
I20260807 11:18:44.560979 15137 heartbeater.cc:464] Registering TS with master...
I20260807 11:18:44.562074 15137 heartbeater.cc:511] Master 127.2.223.190:38653 requested a full tablet report, sending...
I20260807 11:18:44.564889 14910 ts_manager.cc:194] Registered new tserver with Master: a4fea3f1510e49939a41a0fb34d14c54 (127.2.223.129:35445)
I20260807 11:18:44.570071 14910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:44521
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Aug 07 11:18:44 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101523, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:18:44.625979 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:38039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:18:45.316838 15144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:18:45.317317 15144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:18:45.317549 15144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:18:45.326807 15144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:18:45.327026 15144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:18:45.327226 15144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:18:45.339669 15144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.223.148:38039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:18:45.341308 15144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:18:45.342921 15144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:18:45.360867 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:45.360929 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:45.361923 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:18:45.363195 15144 server_base.cc:1034] running on GCE node
I20260807 11:18:45.363893 15144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:18:45.365077 15144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:18:45.366348 15144 hybrid_clock.cc:648] HybridClock initialized: now 1786101525366264 us; error 56 us; skew 500 ppm
I20260807 11:18:45.366793 15144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:18:45 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:18:45.371377 15144 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:18:45.373301 15144 webserver.cc:460] Webserver started at http://127.2.223.130:33147/ using document root <none> and password file <none>
I20260807 11:18:45.374083 15144 fs_manager.cc:359] Metadata directory not provided
I20260807 11:18:45.374245 15144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:18:45.374701 15144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:18:45 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:45.394704 15144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "ac09689849a244d18fcd300b3f7d35c0"
format_stamp: "Formatted at 2026-08-07 11:18:45 on dist-test-slave-5l3k"
server_key: "f98f340e67528b2023bf165bdbc2ccda"
server_key_iv: "d4e65be5e0036a89de836e337a42f9b0"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:45.396217 15144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "ac09689849a244d18fcd300b3f7d35c0"
format_stamp: "Formatted at 2026-08-07 11:18:45 on dist-test-slave-5l3k"
server_key: "f98f340e67528b2023bf165bdbc2ccda"
server_key_iv: "d4e65be5e0036a89de836e337a42f9b0"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:45.404570 15144 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.006s
I20260807 11:18:45.426702 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:45.428668 15144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:18:45.428865 15144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "ac09689849a244d18fcd300b3f7d35c0"
format_stamp: "Formatted at 2026-08-07 11:18:45 on dist-test-slave-5l3k"
server_key: "f98f340e67528b2023bf165bdbc2ccda"
server_key_iv: "d4e65be5e0036a89de836e337a42f9b0"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:45.429152 15144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:18:45.467037 15144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:18:45.474325 15144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:18:45.475173 15144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:18:45.476936 15144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:18:45.479499 15144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:18:45.479662 15144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:45.479859 15144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:18:45.479954 15144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:45.574183 15137 heartbeater.cc:503] Master 127.2.223.190:38653 was elected leader, sending a full tablet report...
I20260807 11:18:45.616514 15144 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:40475
I20260807 11:18:45.616566 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:40475 every 8 connection(s)
I20260807 11:18:45.618353 15144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:18:45.627485 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 15144
I20260807 11:18:45.627709 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:18:45.628511 2942 external_mini_cluster.cc:1428] Setting key d3a51e244d78a10a09953c71f1e8e6f0
Aug 07 11:18:45 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
I20260807 11:18:45.700984 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:45.622984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:41599 (local address 127.2.223.190:38653)
0807 11:18:45.667252 (+ 44268us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:45.667263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:45.670704 (+ 3441us) server_negotiation.cc:407] Connection header received
0807 11:18:45.673235 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:45.673242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:45.673344 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:45.673458 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:45.676222 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:45.677696 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:45.680917 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:45.681787 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:45.681902 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:45.691866 (+ 9964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:45.691878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:45.691886 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:45.691960 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:45.696791 (+ 4831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:45.697891 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:45.697897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:45.697901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:45.698058 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:45.698682 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:45.698687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:45.698693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:45.699103 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:45.699278 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:45.700361 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0807 11:18:45.700618 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44041,"thread_start_us":47496,"threads_started":1}
I20260807 11:18:45.703056 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:45.624655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:38653 (local address 127.2.223.130:41599)
0807 11:18:45.667842 (+ 43187us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:45.667927 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:45.672851 (+ 4924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:45.673985 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:45.674000 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:45.674943 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:18:45.675960 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:45.675991 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:45.678030 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:45.678035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:45.680743 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:45.680761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:45.682382 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:45.682392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:45.682783 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:45.687519 (+ 4736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:18:45.687568 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:18:45.691540 (+ 3972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:18:45.697040 (+ 5500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:45.697053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:45.697076 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:45.697720 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:45.698241 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:45.698246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:45.698250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:45.698537 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:45.699387 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:18:45.699395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:18:45.699887 (+ 492us) client_negotiation.cc:769] Sending connection context
0807 11:18:45.700962 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0807 11:18:45.701543 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42685,"thread_start_us":34007,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
I20260807 11:18:45.707216 15290 heartbeater.cc:347] Connected to a master server at 127.2.223.190:38653
I20260807 11:18:45.707541 15290 heartbeater.cc:464] Registering TS with master...
I20260807 11:18:45.708424 15290 heartbeater.cc:511] Master 127.2.223.190:38653 requested a full tablet report, sending...
I20260807 11:18:45.710348 14910 ts_manager.cc:194] Registered new tserver with Master: ac09689849a244d18fcd300b3f7d35c0 (127.2.223.130:40475)
I20260807 11:18:45.711624 14910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:41599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
W20260807 11:18:45.741539 2942 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.223.131: real 0.109s user 0.003s sys 0.030s
I20260807 11:18:45.742326 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:38039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:18:46.371508 15296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:18:46.371966 15296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:18:46.372232 15296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:18:46.382206 15296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:18:46.382454 15296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:18:46.382790 15296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:18:46.396625 15296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.223.148:38039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:18:46.398205 15296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:18:46.399622 15296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:18:46.425679 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:46.426102 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:18:46.426627 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:18:46.427685 15296 server_base.cc:1034] running on GCE node
I20260807 11:18:46.428426 15296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:18:46.429498 15296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:18:46.430729 15296 hybrid_clock.cc:648] HybridClock initialized: now 1786101526430664 us; error 50 us; skew 500 ppm
I20260807 11:18:46.431116 15296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:18:46 dist-test-slave-5l3k krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:18:46.436554 15296 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:18:46.439783 15296 webserver.cc:460] Webserver started at http://127.2.223.131:35287/ using document root <none> and password file <none>
I20260807 11:18:46.441000 15296 fs_manager.cc:359] Metadata directory not provided
I20260807 11:18:46.441188 15296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:18:46.441756 15296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:18:46 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:18:46.464663 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "03df08688c4c466a8b4789a4cc985722"
format_stamp: "Formatted at 2026-08-07 11:18:46 on dist-test-slave-5l3k"
server_key: "5a26eaa22dadaf3da8d8e9e16c6e2b51"
server_key_iv: "c6a4c510cd46786e134a8688cb7579aa"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:46.466305 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "03df08688c4c466a8b4789a4cc985722"
format_stamp: "Formatted at 2026-08-07 11:18:46 on dist-test-slave-5l3k"
server_key: "5a26eaa22dadaf3da8d8e9e16c6e2b51"
server_key_iv: "c6a4c510cd46786e134a8688cb7579aa"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:46.476555 15296 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.002s
I20260807 11:18:46.500067 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:46.502255 15296 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260807 11:18:46.502597 15296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "03df08688c4c466a8b4789a4cc985722"
format_stamp: "Formatted at 2026-08-07 11:18:46 on dist-test-slave-5l3k"
server_key: "5a26eaa22dadaf3da8d8e9e16c6e2b51"
server_key_iv: "c6a4c510cd46786e134a8688cb7579aa"
server_key_version: "kuduclusterkey@0"
I20260807 11:18:46.502954 15296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:18:46.527081 15296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:18:46.533372 15296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:18:46.533809 15296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:18:46.535718 15296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:18:46.538233 15296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:18:46.538359 15296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:18:46.538614 15296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:18:46.538754 15296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 11:18:46.644111 15296 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:38155
I20260807 11:18:46.644213 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:38155 every 8 connection(s)
I20260807 11:18:46.645740 15296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:18:46.650238 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 15296
I20260807 11:18:46.650439 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:18:46.651093 2942 external_mini_cluster.cc:1428] Setting key 700cc0880787851782f2c3cb4644017b
Aug 07 11:18:46 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:18:46.674723 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.649571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:34129 (local address 127.2.223.190:38653)
0807 11:18:46.650368 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:46.650384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:46.652569 (+ 2185us) server_negotiation.cc:407] Connection header received
0807 11:18:46.655685 (+ 3116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:46.655691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:46.655780 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:46.655897 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:46.658131 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.659423 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.661419 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.662024 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.662176 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.667169 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:46.667178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:46.667182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:46.667241 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:46.671376 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:46.672176 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:46.672179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:46.672182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:46.672341 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:46.672849 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:46.672852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:46.672854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:46.673249 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:46.673405 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:46.674099 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 11:18:46.674296 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":340,"threads_started":1}
I20260807 11:18:46.675747 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.650458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:38653 (local address 127.2.223.131:34129)
0807 11:18:46.651975 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:46.652015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:46.655373 (+ 3358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:46.656188 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:46.656196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:46.656930 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:18:46.657953 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.657974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.659592 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.659596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.661236 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.661248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.662415 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.662425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.662731 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.663642 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:18:46.663665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:18:46.666868 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:18:46.671505 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:46.671512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:46.671526 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:46.672069 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:46.672437 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:18:46.672440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:18:46.672442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:18:46.672748 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:18:46.673362 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:18:46.673367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:18:46.673723 (+ 356us) client_negotiation.cc:769] Sending connection context
0807 11:18:46.673966 (+ 243us) client_negotiation.cc:240] Negotiation successful
0807 11:18:46.674426 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":801,"threads_started":1}
I20260807 11:18:46.677500 15431 heartbeater.cc:347] Connected to a master server at 127.2.223.190:38653
I20260807 11:18:46.677843 15431 heartbeater.cc:464] Registering TS with master...
I20260807 11:18:46.678989 15431 heartbeater.cc:511] Master 127.2.223.190:38653 requested a full tablet report, sending...
I20260807 11:18:46.681128 14909 ts_manager.cc:194] Registered new tserver with Master: 03df08688c4c466a8b4789a4cc985722 (127.2.223.131:38155)
I20260807 11:18:46.682404 14909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:34129
I20260807 11:18:46.693341 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:18:46.715480 15290 heartbeater.cc:503] Master 127.2.223.190:38653 was elected leader, sending a full tablet report...
I20260807 11:18:46.718935 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.699850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.2.223.190:38653)
0807 11:18:46.700083 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:46.700089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:46.701168 (+ 1079us) server_negotiation.cc:407] Connection header received
0807 11:18:46.701356 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:46.701359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:46.701453 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:46.701583 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:46.703712 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.704750 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.706725 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.707442 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.707567 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.711225 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:46.711230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:46.711233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:46.711265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:46.715165 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:46.716189 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:46.716195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:46.716198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:46.716445 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:46.717033 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:46.717041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:46.717043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:46.717554 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:46.717848 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:46.718175 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 11:18:46.718398 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:18:46.723519 14909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:18:46.734535 14909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:18:46.777379 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.758135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.2.223.129:35445)
0807 11:18:46.759569 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:46.759578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:46.763621 (+ 4043us) server_negotiation.cc:407] Connection header received
0807 11:18:46.763749 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:46.763755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:46.764148 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:46.764399 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:18:46.766489 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.769288 (+ 2799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.774017 (+ 4729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.775602 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.775789 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.776222 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:46.776491 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 11:18:46.776740 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":872,"threads_started":1}
I20260807 11:18:46.778271 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.762890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:38155 (local address 127.0.0.1:40362)
0807 11:18:46.764450 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:46.764468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:46.764749 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:46.766939 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:46.766958 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:46.767008 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:18:46.767931 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.767946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.770263 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.770267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.773633 (+ 3366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.773645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.776853 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.776862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.777242 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.777307 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 11:18:46.777399 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 11:18:46.777556 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":1,"thread_start_us":1062,"threads_started":1}
I20260807 11:18:46.777588 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.758613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:35445 (local address 127.0.0.1:48382)
0807 11:18:46.763276 (+ 4663us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:46.763296 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:46.763457 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:46.764750 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:46.764758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:46.764802 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:18:46.766226 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.766239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.769460 (+ 3221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.769465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.773859 (+ 4394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.773871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.775899 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.775906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.776231 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.776293 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 11:18:46.776444 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 11:18:46.776699 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"mutex_wait_us":101,"thread_start_us":791,"threads_started":1}
I20260807 11:18:46.779250 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.763569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.2.223.131:38155)
0807 11:18:46.766053 (+ 2484us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:46.766064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:46.766134 (+ 70us) server_negotiation.cc:407] Connection header received
0807 11:18:46.766262 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:46.766272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:46.766641 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:46.767975 (+ 1334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:18:46.768470 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.770055 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.773941 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.776462 (+ 2521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.777791 (+ 1329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.778122 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:46.778204 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 11:18:46.778452 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2210,"thread_start_us":1983,"threads_started":1}
I20260807 11:18:46.779098 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.758002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40475 (local address 127.0.0.1:37184)
0807 11:18:46.760390 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:46.760450 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:46.760672 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:46.764234 (+ 3562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:46.764253 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:46.764284 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:18:46.765956 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.765972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.768887 (+ 2915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.768895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.773538 (+ 4643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:46.773577 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.777385 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.777394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:46.777623 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.777664 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 11:18:46.777791 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 11:18:46.778294 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2080,"mutex_wait_us":100,"thread_start_us":3304,"threads_started":1}
I20260807 11:18:46.780176 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:46.758536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.2.223.130:40475)
0807 11:18:46.761917 (+ 3381us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:46.761929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:46.761967 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:18:46.762101 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:46.762110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:46.762377 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:46.762695 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:18:46.766441 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.768728 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.773791 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:46.776311 (+ 2520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:46.776590 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:46.776969 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:46.779313 (+ 2344us) server_negotiation.cc:299] Negotiation successful
0807 11:18:46.779538 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3118,"thread_start_us":2743,"threads_started":1}
I20260807 11:18:46.783639 15362 tablet_service.cc:1467] Processing CreateTablet for tablet cc32151db28143c3bd36bf89ef8846d7 (DEFAULT_TABLE table=test-table [id=3f76e0f72513420ea7b1e947d5263bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:18:46.783818 15070 tablet_service.cc:1467] Processing CreateTablet for tablet cc32151db28143c3bd36bf89ef8846d7 (DEFAULT_TABLE table=test-table [id=3f76e0f72513420ea7b1e947d5263bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:18:46.785434 15362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32151db28143c3bd36bf89ef8846d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:18:46.784664 15220 tablet_service.cc:1467] Processing CreateTablet for tablet cc32151db28143c3bd36bf89ef8846d7 (DEFAULT_TABLE table=test-table [id=3f76e0f72513420ea7b1e947d5263bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:18:46.786319 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32151db28143c3bd36bf89ef8846d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:18:46.786883 15220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32151db28143c3bd36bf89ef8846d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:18:46.802198 15451 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Bootstrap starting.
I20260807 11:18:46.807775 15451 tablet_bootstrap.cc:654] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Neither blocks nor log segments found. Creating new log.
I20260807 11:18:46.809913 15451 log.cc:824] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Log is configured to *not* fsync() on all Append() calls
I20260807 11:18:46.815254 15452 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Bootstrap starting.
I20260807 11:18:46.815763 15451 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: No bootstrap required, opened a new log
I20260807 11:18:46.816105 15451 ts_tablet_manager.cc:1397] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260807 11:18:46.819393 15453 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Bootstrap starting.
I20260807 11:18:46.821106 15452 tablet_bootstrap.cc:654] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Neither blocks nor log segments found. Creating new log.
I20260807 11:18:46.823264 15452 log.cc:824] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Log is configured to *not* fsync() on all Append() calls
I20260807 11:18:46.826387 15453 tablet_bootstrap.cc:654] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Neither blocks nor log segments found. Creating new log.
I20260807 11:18:46.827646 15452 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: No bootstrap required, opened a new log
I20260807 11:18:46.828016 15452 ts_tablet_manager.cc:1397] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260807 11:18:46.828775 15453 log.cc:824] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:18:46.832230 15451 raft_consensus.cc:348] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.832849 15451 raft_consensus.cc:374] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:18:46.833025 15451 raft_consensus.cc:729] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03df08688c4c466a8b4789a4cc985722, State: Initialized, Role: FOLLOWER
I20260807 11:18:46.833873 15453 tablet_bootstrap.cc:492] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: No bootstrap required, opened a new log
I20260807 11:18:46.833860 15451 consensus_queue.cc:260] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.834254 15453 ts_tablet_manager.cc:1397] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260807 11:18:46.838148 15431 heartbeater.cc:503] Master 127.2.223.190:38653 was elected leader, sending a full tablet report...
I20260807 11:18:46.839005 15451 ts_tablet_manager.cc:1428] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Time spent starting tablet: real 0.023s user 0.011s sys 0.013s
I20260807 11:18:46.843586 15452 raft_consensus.cc:348] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.844458 15452 raft_consensus.cc:374] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:18:46.844694 15452 raft_consensus.cc:729] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fea3f1510e49939a41a0fb34d14c54, State: Initialized, Role: FOLLOWER
I20260807 11:18:46.845490 15452 consensus_queue.cc:260] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.848147 15452 ts_tablet_manager.cc:1428] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Time spent starting tablet: real 0.020s user 0.018s sys 0.003s
I20260807 11:18:46.849529 15453 raft_consensus.cc:348] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.850101 15453 raft_consensus.cc:374] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:18:46.850265 15453 raft_consensus.cc:729] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac09689849a244d18fcd300b3f7d35c0, State: Initialized, Role: FOLLOWER
I20260807 11:18:46.851147 15453 consensus_queue.cc:260] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:46.853828 15453 ts_tablet_manager.cc:1428] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Time spent starting tablet: real 0.019s user 0.004s sys 0.015s
W20260807 11:18:46.898787 15432 tablet.cc:2369] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:18:46.923182 15291 tablet.cc:2369] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:18:47.050650 15138 tablet.cc:2369] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:18:47.106343 15457 raft_consensus.cc:482] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:18:47.106873 15457 raft_consensus.cc:504] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:47.109185 15457 leader_election.cc:288] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4fea3f1510e49939a41a0fb34d14c54 (127.2.223.129:35445), ac09689849a244d18fcd300b3f7d35c0 (127.2.223.130:40475)
I20260807 11:18:47.118921 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.110643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:45575 (local address 127.2.223.129:35445)
0807 11:18:47.110924 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.110933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.110962 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:18:47.111057 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.111065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.111167 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.111288 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:18:47.111999 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.112867 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.116238 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.117738 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.117994 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.118094 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.118471 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.118643 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 11:18:47.119045 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.109453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:35445 (local address 127.2.223.131:45575)
0807 11:18:47.110041 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:47.110064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:47.110210 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:47.111332 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:47.111337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:47.111360 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:18:47.111823 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:47.111834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.113020 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.113029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:47.115996 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:47.116012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.117964 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.117974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:47.118235 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.118294 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 11:18:47.118429 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 11:18:47.118649 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":43}
I20260807 11:18:47.119449 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.109683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:39761 (local address 127.2.223.130:40475)
0807 11:18:47.109994 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.110013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.111110 (+ 1097us) server_negotiation.cc:407] Connection header received
0807 11:18:47.111313 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.111318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.111396 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.111525 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:18:47.112934 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.114153 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.116054 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.117973 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.118176 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.118226 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.118930 (+ 704us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.119074 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:18:47.120605 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.109637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40475 (local address 127.2.223.131:39761)
0807 11:18:47.110983 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0807 11:18:47.110998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:18:47.111174 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:18:47.111608 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:18:47.111616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:18:47.111658 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:18:47.112713 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:47.112734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.114336 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.114340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:47.115896 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:18:47.115908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.118472 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.118500 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:18:47.118728 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.118763 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:18:47.119690 (+ 927us) client_negotiation.cc:240] Negotiation successful
0807 11:18:47.119827 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":4,"thread_start_us":969,"threads_started":1}
I20260807 11:18:47.120420 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32151db28143c3bd36bf89ef8846d7" candidate_uuid: "03df08688c4c466a8b4789a4cc985722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fea3f1510e49939a41a0fb34d14c54" is_pre_election: true
I20260807 11:18:47.121199 15090 raft_consensus.cc:2454] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03df08688c4c466a8b4789a4cc985722 in term 0.
I20260807 11:18:47.121873 15312 leader_election.cc:302] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03df08688c4c466a8b4789a4cc985722, a4fea3f1510e49939a41a0fb34d14c54; no voters:
I20260807 11:18:47.122318 15457 raft_consensus.cc:2781] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:18:47.121883 15240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32151db28143c3bd36bf89ef8846d7" candidate_uuid: "03df08688c4c466a8b4789a4cc985722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac09689849a244d18fcd300b3f7d35c0" is_pre_election: true
I20260807 11:18:47.122602 15457 raft_consensus.cc:482] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:18:47.122788 15457 raft_consensus.cc:3037] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:18:47.122752 15240 raft_consensus.cc:2454] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03df08688c4c466a8b4789a4cc985722 in term 0.
I20260807 11:18:47.126663 15457 raft_consensus.cc:504] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:47.127728 15457 leader_election.cc:288] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [CANDIDATE]: Term 1 election: Requested vote from peers a4fea3f1510e49939a41a0fb34d14c54 (127.2.223.129:35445), ac09689849a244d18fcd300b3f7d35c0 (127.2.223.130:40475)
I20260807 11:18:47.128290 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32151db28143c3bd36bf89ef8846d7" candidate_uuid: "03df08688c4c466a8b4789a4cc985722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fea3f1510e49939a41a0fb34d14c54"
I20260807 11:18:47.128595 15090 raft_consensus.cc:3037] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:18:47.128580 15240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32151db28143c3bd36bf89ef8846d7" candidate_uuid: "03df08688c4c466a8b4789a4cc985722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac09689849a244d18fcd300b3f7d35c0"
I20260807 11:18:47.128826 15240 raft_consensus.cc:3037] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:18:47.132897 15090 raft_consensus.cc:2454] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03df08688c4c466a8b4789a4cc985722 in term 1.
I20260807 11:18:47.133550 15312 leader_election.cc:302] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03df08688c4c466a8b4789a4cc985722, a4fea3f1510e49939a41a0fb34d14c54; no voters:
I20260807 11:18:47.133672 15240 raft_consensus.cc:2454] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03df08688c4c466a8b4789a4cc985722 in term 1.
I20260807 11:18:47.134048 15457 raft_consensus.cc:2781] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:18:47.135682 15457 raft_consensus.cc:686] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 1 LEADER]: Becoming Leader. State: Replica: 03df08688c4c466a8b4789a4cc985722, State: Running, Role: LEADER
I20260807 11:18:47.136310 15457 consensus_queue.cc:237] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } }
I20260807 11:18:47.143244 14909 catalog_manager.cc:5495] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03df08688c4c466a8b4789a4cc985722 (127.2.223.131). New cstate: current_term: 1 leader_uuid: "03df08688c4c466a8b4789a4cc985722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03df08688c4c466a8b4789a4cc985722" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 38155 } health_report { overall_health: HEALTHY } } }
I20260807 11:18:47.232116 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.223310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.2.223.131:38155)
0807 11:18:47.223573 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.223583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.224243 (+ 660us) server_negotiation.cc:407] Connection header received
0807 11:18:47.224457 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.224465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.224589 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.224851 (+ 262us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:18:47.225544 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.226996 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.228939 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.229724 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.229894 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.230512 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:18:47.231300 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:18:47.231467 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.231605 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.231713 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:18:47.242988 15240 raft_consensus.cc:1261] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 1 FOLLOWER]: Refusing update from remote peer 03df08688c4c466a8b4789a4cc985722: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:18:47.243144 15090 raft_consensus.cc:1261] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 1 FOLLOWER]: Refusing update from remote peer 03df08688c4c466a8b4789a4cc985722: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:18:47.244179 15461 consensus_queue.cc:1035] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac09689849a244d18fcd300b3f7d35c0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:18:47.244735 15457 consensus_queue.cc:1035] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4fea3f1510e49939a41a0fb34d14c54" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:18:47.288966 14910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:18:47.289289 14910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:18:47.293150 14910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48b2bde6a8b146158c9e2c5e8e9b9e64: Sending DeleteTablet for 3 replicas of tablet cc32151db28143c3bd36bf89ef8846d7
I20260807 11:18:47.294656 15220 tablet_service.cc:1514] Processing DeleteTablet for tablet cc32151db28143c3bd36bf89ef8846d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:18:47 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37184
I20260807 11:18:47.294625 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet cc32151db28143c3bd36bf89ef8846d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:18:47 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:48382
I20260807 11:18:47.295811 15362 tablet_service.cc:1514] Processing DeleteTablet for tablet cc32151db28143c3bd36bf89ef8846d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:18:47 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:40362
I20260807 11:18:47.296070 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 14997
I20260807 11:18:47.299147 15477 tablet_replica.cc:330] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: stopping tablet replica
I20260807 11:18:47.299852 15475 tablet_replica.cc:330] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: stopping tablet replica
I20260807 11:18:47.300177 15476 tablet_replica.cc:330] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: stopping tablet replica
I20260807 11:18:47.300578 15477 raft_consensus.cc:2229] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:18:47.300700 15476 raft_consensus.cc:2229] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:18:47.301265 15475 raft_consensus.cc:2229] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:18:47.301479 15476 raft_consensus.cc:2258] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:18:47.302204 15477 raft_consensus.cc:2258] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:18:47.302736 15475 raft_consensus.cc:2258] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:18:47.304289 15476 ts_tablet_manager.cc:1905] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:18:47.305401 15477 ts_tablet_manager.cc:1905] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:18:47.305979 15475 ts_tablet_manager.cc:1905] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:18:47.311955 15476 ts_tablet_manager.cc:1918] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:18:47.312116 15476 log.cc:1196] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/cc32151db28143c3bd36bf89ef8846d7
I20260807 11:18:47.313043 15476 ts_tablet_manager.cc:1939] T cc32151db28143c3bd36bf89ef8846d7 P a4fea3f1510e49939a41a0fb34d14c54: Deleting consensus metadata
I20260807 11:18:47.313470 15477 ts_tablet_manager.cc:1918] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:18:47.313715 15477 log.cc:1196] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/cc32151db28143c3bd36bf89ef8846d7
I20260807 11:18:47.314611 15477 ts_tablet_manager.cc:1939] T cc32151db28143c3bd36bf89ef8846d7 P 03df08688c4c466a8b4789a4cc985722: Deleting consensus metadata
I20260807 11:18:47.315315 14893 catalog_manager.cc:4841] TS a4fea3f1510e49939a41a0fb34d14c54 (127.2.223.129:35445): tablet cc32151db28143c3bd36bf89ef8846d7 (table test-table [id=3f76e0f72513420ea7b1e947d5263bc0]) successfully deleted
I20260807 11:18:47.316965 14894 catalog_manager.cc:4841] TS 03df08688c4c466a8b4789a4cc985722 (127.2.223.131:38155): tablet cc32151db28143c3bd36bf89ef8846d7 (table test-table [id=3f76e0f72513420ea7b1e947d5263bc0]) successfully deleted
I20260807 11:18:47.317555 15475 ts_tablet_manager.cc:1918] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:18:47.317821 15475 log.cc:1196] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/cc32151db28143c3bd36bf89ef8846d7
I20260807 11:18:47.318470 15475 ts_tablet_manager.cc:1939] T cc32151db28143c3bd36bf89ef8846d7 P ac09689849a244d18fcd300b3f7d35c0: Deleting consensus metadata
I20260807 11:18:47.320425 14894 catalog_manager.cc:4841] TS ac09689849a244d18fcd300b3f7d35c0 (127.2.223.130:40475): tablet cc32151db28143c3bd36bf89ef8846d7 (table test-table [id=3f76e0f72513420ea7b1e947d5263bc0]) successfully deleted
Aug 07 11:18:47 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:18:47.337311 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.297153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.2.223.129:35445)
0807 11:18:47.297365 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.297375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.318260 (+ 20885us) server_negotiation.cc:407] Connection header received
0807 11:18:47.318516 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.318522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.318608 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.318764 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:47.320178 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.321450 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.323302 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.323890 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.324060 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.329530 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:47.329542 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:47.329556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:47.329588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:47.333654 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.334388 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.334395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.334397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.334634 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.335108 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.335113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.335116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.335678 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:47.336175 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.336414 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.336761 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:18:47.338256 15132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:18:47.472051 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 14997
I20260807 11:18:47.494277 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 15144
Aug 07 11:18:47 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:18:47.512496 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.495246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.2.223.130:40475)
0807 11:18:47.495472 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.495478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.495511 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:18:47.495638 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.495644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.495763 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.495874 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:47.497173 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.498291 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.499857 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.500493 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.500631 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.505041 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:47.505062 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:47.505084 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:47.505135 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:47.508938 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.509981 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.509991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.509995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.510154 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.510659 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.510664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.510668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.511162 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:47.511462 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.511743 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.512080 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:18:47.513296 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:18:47.604830 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 15144
I20260807 11:18:47.626547 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 15296
Aug 07 11:18:47 dist-test-slave-5l3k krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:18:47.645555 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:18:47.627421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.2.223.131:38155)
0807 11:18:47.627673 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:18:47.627680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:18:47.627703 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:18:47.627855 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:18:47.627862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:18:47.627956 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:18:47.628074 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:18:47.629447 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.630830 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.632209 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:18:47.632924 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:18:47.633100 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:18:47.637693 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:18:47.637709 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:18:47.637746 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:18:47.637791 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:18:47.642171 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.643079 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.643089 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.643093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.643315 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:18:47.643770 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:18:47.643775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:18:47.643779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:18:47.644253 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:18:47.644624 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 11:18:47.644886 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 11:18:47.645128 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 11:18:47.646461 15426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:18:47.771476 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 15296
I20260807 11:18:47.795707 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 14876
I20260807 11:18:47.796952 14944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:18:47.914698 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 14876
I20260807 11:18:47.942077 2942 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 11:18:48.287158 2942 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 11:18:48.287518 2942 mini_ranger.cc:66] Stopping Ranger...
I20260807 11:18:48.643020 2942 mini_ranger.cc:68] Stopped Ranger
2026-08-07 11:18:48.643 UTC [14441] LOG: received smart shutdown request
2026-08-07 11:18:48.648 UTC [14441] LOG: background worker "logical replication launcher" (PID 14451) exited with exit code 1
2026-08-07 11:18:48.648 UTC [14445] LOG: shutting down
2026-08-07 11:18:48.826 UTC [14441] LOG: database system is shut down
2026-08-07T11:18:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53580 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:18:49 dist-test-slave-5l3k krb5kdc[15492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:18:49 dist-test-slave-5l3k krb5kdc[15492](info): set up 2 sockets
Aug 07 11:18:49 dist-test-slave-5l3k krb5kdc[15492](info): commencing operation
krb5kdc: starting...
W20260807 11:18:50.125023 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:18:50 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:18:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangeradmin_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangeradmin_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangerlookup_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangerlookup_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
I20260807 11:18:50.547132 2942 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgCuO6z/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/postgres -l logfile start
2026-08-07 11:18:53.108 UTC [15526] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 11:18:53.108 UTC [15526] LOG: listening on IPv4 address "127.2.223.148", port 54327
2026-08-07 11:18:53.112 UTC [15526] LOG: listening on Unix socket "/tmp/.s.PGSQL.54327"
2026-08-07 11:18:53.133 UTC [15528] LOG: database system was shut down at 2026-08-07 11:18:51 UTC
2026-08-07 11:18:53.136 UTC [15526] LOG: database system is ready to accept connections
I20260807 11:18:54.149290 2942 mini_postgres.cc:96] Postgres bound to 54327
2026-08-07 11:18:54.176 UTC [15538] FATAL: database "slave" does not exist
127.2.223.148:54327 - accepting connections
I20260807 11:18:54.177341 2942 mini_ranger.cc:161] Starting Ranger...
I20260807 11:18:54.210197 2942 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 11:18:54.681191 2942 mini_ranger.cc:87] Created ranger Postgres database
I20260807 11:18:54.681321 2942 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin
2026-08-07 11:18:54,969 [I] DB FLAVOR :POSTGRES
2026-08-07 11:18:54,969 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:18:54,969 [I] Checking connection..
2026-08-07 11:18:54,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 11:18:55,212 [I] Checking connection passed.
2026-08-07 11:18:55,213 [I] --------- Verifying version history table ---------
2026-08-07 11:18:55,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:55,482 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:18:55,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:55,753 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:18:55,753 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 11:18:55,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 11:18:56.010 UTC [15604] WARNING: there is no transaction in progress
2026-08-07 11:18:56,018 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 11:18:56,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:18:56,295 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 11:18:56,295 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:18:56,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 11:18:56,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 11:18:56,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5l3k.c.gcp-upstream.internal','N') ;"
2026-08-07 11:18:57,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:18:57,366 [I] Table x_portal_user does not exist in database ranger
2026-08-07 11:18:57,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:18:57,635 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 11:18:57,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:18:57,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:18:58,145 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 11:18:58,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 11:18:59.096 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.111 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.126 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.140 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.155 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.171 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.225 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.232 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.245 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.255 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.265 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.278 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.288 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.294 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.301 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59.307 UTC [15718] WARNING: there is no transaction in progress
2026-08-07 11:18:59,944 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 11:18:59,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 11:19:00,200 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 11:19:00,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:19:00,470 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 11:19:00,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:19:00,736 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 11:19:00,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:19:01,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:19:01,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5l3k.c.gcp-upstream.internal';"
2026-08-07 11:19:01,519 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 11:19:01,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 11:19:01,790 [I] DB_PATCHES have already been applied
I20260807 11:19:01.797348 2942 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgCuO6z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 11:19:01.797446 2942 mini_ranger.cc:193] Using host: 127.2.223.148
I20260807 11:19:01.822741 2942 mini_ranger.cc:227] Ranger admin URL: http://127.2.223.148:57591
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 11:19:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 11:19:03.035956 2942 mini_ranger.cc:160] Time spent starting Ranger: real 8.859s user 0.009s sys 0.062s
Aug 07, 2026 11:19:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 11:19:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 11:19:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 11:19:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 11:19:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 11:19:27.243443 2942 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.223.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.223.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangerkms_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.223.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/rangerkms_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.223.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.223.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.223.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/HTTP_127.2.223.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 11:19:27 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 11:19:27.463137 2942 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 11:19:27.491988 2942 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 11:19:28.375818 2942 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 11:19:28.375952 2942 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms
I20260807 11:19:28.376009 2942 mini_ranger_kms.cc:227] Using postgres at 127.2.223.148:54327
2026-08-07 11:19:28,789 [I] DB FLAVOR :POSTGRES
2026-08-07 11:19:28,790 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:19:28,790 [I] Checking connection
2026-08-07 11:19:28,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 11:19:29,062 [I] connection success
2026-08-07 11:19:29,062 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 11:19:29,062 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 11:19:29,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 11:19:29,308 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 11:19:29,309 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:19:29,309 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 11:19:29,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.223.148:54327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 11:19:29,594 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 11:19:30.046909 2942 mini_ranger_kms.cc:310] Created kms service
I20260807 11:19:30.249655 2942 mini_ranger_kms.cc:326] Created kudu user
I20260807 11:19:30.298954 2942 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 11:19:30 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:30.723043 2942 mini_ranger_kms.cc:384] Added ranger policy
I20260807 11:19:30.723398 2942 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgCuO6z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgCuO6z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 11:19:30.723461 2942 mini_ranger_kms.cc:240] Using host: 127.2.223.148
I20260807 11:19:30.748131 2942 mini_ranger_kms.cc:276] Ranger KMS PID: 15910
I20260807 11:19:30.748260 2942 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.223.148:50527
Aug 07, 2026 11:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 11:19:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 11:19:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 11:19:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 11:19:32 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786101572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:33.140290 2942 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.677s user 0.083s sys 0.358s
I20260807 11:19:33.140550 2942 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 11:19:33.140733 2942 mini_ranger_kms.cc:401] 127.2.223.148:50527/kms/v1/keys
Aug 07 11:19:33 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
Aug 07 11:19:33 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.223.148@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
Aug 07 11:19:36 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:19:36.163411 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33615
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33615
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:36.520598 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:36.520877 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:36.520977 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:36.528177 15962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:36.528275 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:36.528321 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:36.528378 15962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:36.528425 15962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:36.538326 15962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.223.148:50527/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33615
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:36.539485 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:36.540741 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:36.552278 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:36.552366 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:36.552318 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:36.553563 15962 server_base.cc:1034] running on GCE node
I20260807 11:19:36.554361 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:36.555738 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:36.556967 15962 hybrid_clock.cc:648] HybridClock initialized: now 1786101576556928 us; error 57 us; skew 500 ppm
I20260807 11:19:36.557339 15962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:36 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:36.568192 15962 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:36.569839 15962 webserver.cc:460] Webserver started at http://127.2.223.190:38947/ using document root <none> and password file <none>
I20260807 11:19:36.570593 15962 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:36.570737 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:36.571069 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:19:36 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:36.627512 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "325be22e4d2c483a8825aa6019d0bbc1"
format_stamp: "Formatted at 2026-08-07 11:19:36 on dist-test-slave-5l3k"
server_key: "04afe5b1daffd11c8243b7417592b8c9"
server_key_iv: "7decaaa16fc1365e8157d4ad122ebf88"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:36.628660 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "325be22e4d2c483a8825aa6019d0bbc1"
format_stamp: "Formatted at 2026-08-07 11:19:36 on dist-test-slave-5l3k"
server_key: "04afe5b1daffd11c8243b7417592b8c9"
server_key_iv: "7decaaa16fc1365e8157d4ad122ebf88"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:36.635810 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260807 11:19:36.657490 15978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:36.659260 15962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:19:36.659451 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "325be22e4d2c483a8825aa6019d0bbc1"
format_stamp: "Formatted at 2026-08-07 11:19:36 on dist-test-slave-5l3k"
server_key: "04afe5b1daffd11c8243b7417592b8c9"
server_key_iv: "7decaaa16fc1365e8157d4ad122ebf88"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:36.659725 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:36.681247 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:36.685091 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:36.685370 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:36.704849 15962 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33615
I20260807 11:19:36.704870 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33615 every 8 connection(s)
I20260807 11:19:36.706339 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:36.707958 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 15962
I20260807 11:19:36.708197 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:19:36.708922 2942 external_mini_cluster.cc:1428] Setting key 2e85cf9bf0d5fb36a8699d6b5fb892e3
I20260807 11:19:36.710847 16032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:19:36 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:36.724568 16032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Bootstrap starting.
I20260807 11:19:36.728884 16032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:36.730181 16032 log.cc:824] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:36.733875 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:36.711079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.2.223.190:33615)
0807 11:19:36.711914 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:36.711929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:36.711979 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:19:36.714074 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:36.714102 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:36.714654 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:36.715100 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:36.716603 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:36.718704 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:36.719920 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:36.720951 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:36.721168 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:36.725076 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:36.725088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:36.725107 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:36.725155 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:36.729183 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:36.729866 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:36.729875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:36.729883 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:36.730087 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:36.730525 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:36.730545 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:36.730548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:36.731249 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:36.731481 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:36.732157 (+ 676us) server_negotiation.cc:299] Negotiation successful
0807 11:19:36.732570 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260807 11:19:36.733899 16032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: No bootstrap required, opened a new log
I20260807 11:19:36.742033 16032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:36.742380 16032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:36.742565 16032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325be22e4d2c483a8825aa6019d0bbc1, State: Initialized, Role: FOLLOWER
I20260807 11:19:36.743054 16032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:36.743250 16032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:19:36.743346 16032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:19:36.743466 16032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:36.745507 16032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:36.746089 16032 leader_election.cc:302] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325be22e4d2c483a8825aa6019d0bbc1; no voters:
I20260807 11:19:36.746804 16032 leader_election.cc:288] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:19:36.746918 16038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:36.747689 16038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 LEADER]: Becoming Leader. State: Replica: 325be22e4d2c483a8825aa6019d0bbc1, State: Running, Role: LEADER
I20260807 11:19:36.748232 16038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:36.748965 16032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:36.751165 16039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325be22e4d2c483a8825aa6019d0bbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } } }
I20260807 11:19:36.751137 16040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325be22e4d2c483a8825aa6019d0bbc1. Latest consensus state: current_term: 1 leader_uuid: "325be22e4d2c483a8825aa6019d0bbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } } }
I20260807 11:19:36.751565 16039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:36.751622 16040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:36.753360 16045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:36.758913 16045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:36.759709 16032 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 11:19:36.784924 16045 catalog_manager.cc:1347] Generated new cluster ID: 07548abe66674c3bb80775aa721837bc
I20260807 11:19:36.785135 16045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:36.804356 16045 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:19:36.806388 16045 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:36.819239 16045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Generated new TSK 0
I20260807 11:19:36.820392 16045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:19:37.626971 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:37 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786101577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 11:19:38.052362 16079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:38.052670 16079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:38.052838 16079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:38.060626 16079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:38.060782 16079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:38.060986 16079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:19:38.072921 16079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:38.074771 16079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:38.076225 16079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:38.097128 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:38.099133 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:38.103557 16079 server_base.cc:1034] running on GCE node
W20260807 11:19:38.104001 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:38.104832 16079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:38.107432 16079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:38.108685 16079 hybrid_clock.cc:648] HybridClock initialized: now 1786101578108609 us; error 70 us; skew 500 ppm
I20260807 11:19:38.109205 16079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:38 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101578, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:38.113395 16079 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:19:38.116321 16079 webserver.cc:460] Webserver started at http://127.2.223.129:35729/ using document root <none> and password file <none>
I20260807 11:19:38.117063 16079 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:38.117235 16079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:38.117560 16079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:19:38 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101578, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:38.139200 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc"
format_stamp: "Formatted at 2026-08-07 11:19:38 on dist-test-slave-5l3k"
server_key: "16fa76590a725fea93aa02ceec62bf0b"
server_key_iv: "6eb4a803e0b892365ad0536061b05ea7"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:38.140312 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc"
format_stamp: "Formatted at 2026-08-07 11:19:38 on dist-test-slave-5l3k"
server_key: "16fa76590a725fea93aa02ceec62bf0b"
server_key_iv: "6eb4a803e0b892365ad0536061b05ea7"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:38.150252 16079 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260807 11:19:38.183197 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:38.185276 16079 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.014s sys 0.000s
I20260807 11:19:38.185482 16079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc"
format_stamp: "Formatted at 2026-08-07 11:19:38 on dist-test-slave-5l3k"
server_key: "16fa76590a725fea93aa02ceec62bf0b"
server_key_iv: "6eb4a803e0b892365ad0536061b05ea7"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:38.185784 16079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:38.225728 16079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:38.230115 16079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:38.230460 16079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:38.231757 16079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:38.233347 16079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:38.233453 16079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:38.233590 16079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:38.233688 16079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:19:38 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:38.449808 16079 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:44443
I20260807 11:19:38.449846 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:44443 every 8 connection(s)
I20260807 11:19:38.451598 16079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:19:38.457134 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16079
I20260807 11:19:38.457357 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:19:38.458209 2942 external_mini_cluster.cc:1428] Setting key 3cd05c73205875c0b98028e4c6489521
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Aug 07 11:19:38 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101578, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:38.540225 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:38.456316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.129:34473)
0807 11:19:38.485388 (+ 29072us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:38.485469 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:38.488394 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:38.501241 (+ 12847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:38.501258 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:38.515874 (+ 14616us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:38.516904 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:38.516940 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:38.523944 (+ 7004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:38.523949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:38.525321 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:38.525337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:38.526715 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:38.526728 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:38.527039 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:38.527879 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:38.527922 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:38.530779 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:38.535237 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:38.535247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:38.535268 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:38.535859 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:38.536349 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:38.536354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:38.536357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:38.536630 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:38.537373 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:38.537381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:38.537868 (+ 487us) client_negotiation.cc:769] Sending connection context
0807 11:19:38.538191 (+ 323us) client_negotiation.cc:240] Negotiation successful
0807 11:19:38.538758 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28636,"thread_start_us":32445,"threads_started":1}
I20260807 11:19:38.543336 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:38.455628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:34473 (local address 127.2.223.190:33615)
0807 11:19:38.500225 (+ 44597us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:38.500235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:38.500268 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:19:38.500352 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:38.500358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:38.500497 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:38.500649 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:38.517217 (+ 16568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:38.523711 (+ 6494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:38.525510 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:38.526148 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:38.526302 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:38.531109 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:38.531116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:38.531121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:38.531167 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:38.535058 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:38.536019 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:38.536026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:38.536030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:38.536198 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:38.536778 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:38.536782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:38.536786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:38.537150 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:38.537285 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:38.542717 (+ 5432us) server_negotiation.cc:299] Negotiation successful
0807 11:19:38.542967 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44437,"thread_start_us":44240,"threads_started":1}
I20260807 11:19:38.545305 16218 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:38.545653 16218 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:38.546526 16218 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:38.549116 15996 ts_manager.cc:194] Registered new tserver with Master: ff33342e1e4340cf8b35fb4ad3a9d5bc (127.2.223.129:44443)
I20260807 11:19:38.551323 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:34473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
W20260807 11:19:38.569725 2942 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.223.130: real 0.108s user 0.002s sys 0.026s
I20260807 11:19:38.570549 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:38 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101577, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
W20260807 11:19:39.050009 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:39.050400 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:39.050603 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:39.062166 16225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:39.062299 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:39.062505 16225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:19:39.078100 16225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:39.079847 16225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:39.081547 16225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:39.097406 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:39.097379 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:39.097561 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:39.098120 16225 server_base.cc:1034] running on GCE node
I20260807 11:19:39.098861 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:39.099797 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:39.100993 16225 hybrid_clock.cc:648] HybridClock initialized: now 1786101579100957 us; error 29 us; skew 500 ppm
I20260807 11:19:39.101274 16225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:39.104584 16225 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:19:39.105986 16225 webserver.cc:460] Webserver started at http://127.2.223.130:40541/ using document root <none> and password file <none>
I20260807 11:19:39.106745 16225 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:39.106875 16225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:39.107185 16225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:39.124591 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "e0c7e2cdf2eb457db9a026353414b0e3"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "9e04e3aa9f73407cd07893fa7bc6282c"
server_key_iv: "2d3b2d25e89a44cefdbe429a1a537685"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.125597 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "e0c7e2cdf2eb457db9a026353414b0e3"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "9e04e3aa9f73407cd07893fa7bc6282c"
server_key_iv: "2d3b2d25e89a44cefdbe429a1a537685"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.132282 16225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:19:39.149710 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:39.151494 16225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260807 11:19:39.151688 16225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "e0c7e2cdf2eb457db9a026353414b0e3"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "9e04e3aa9f73407cd07893fa7bc6282c"
server_key_iv: "2d3b2d25e89a44cefdbe429a1a537685"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.151937 16225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:39.175922 16225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:39.179641 16225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:39.179893 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:39.180821 16225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:39.182145 16225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:39.182228 16225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:39.182312 16225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:39.182356 16225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:39.217799 16225 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42039
I20260807 11:19:39.217839 16372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42039 every 8 connection(s)
I20260807 11:19:39.219228 16225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:19:39.225481 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16225
I20260807 11:19:39.225677 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:19:39.226118 2942 external_mini_cluster.cc:1428] Setting key b42ec980b5596a56fa52b9d051ec0206
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:39.239547 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:39.221891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:35225 (local address 127.2.223.190:33615)
0807 11:19:39.222430 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:39.222434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:39.223242 (+ 808us) server_negotiation.cc:407] Connection header received
0807 11:19:39.225035 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:39.225042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:39.225186 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:39.225316 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:39.226926 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.228054 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.229541 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.229969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.230121 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:39.233541 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:39.233546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:39.233549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:39.233581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:39.236431 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.237127 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.237133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.237136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.237267 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.237746 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.237751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.237753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.238031 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:39.238189 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:39.238932 (+ 743us) server_negotiation.cc:299] Negotiation successful
0807 11:19:39.239163 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260807 11:19:39.240450 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:39.222305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.130:35225)
0807 11:19:39.223071 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:39.223115 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:39.224771 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:39.225541 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:39.225549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:39.226067 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:39.226746 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:39.226766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.228214 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.228218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:39.229396 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:39.229408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.230226 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.230234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:39.230465 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:39.231298 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:39.231325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:39.233332 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:39.236573 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:39.236582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:39.236597 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:39.236999 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:39.237380 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:39.237385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:39.237389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:39.237600 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:39.238162 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:39.238170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:39.238541 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 11:19:39.238814 (+ 273us) client_negotiation.cc:240] Negotiation successful
0807 11:19:39.239303 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":353,"threads_started":1}
I20260807 11:19:39.242046 16373 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:39.242410 16373 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:39.243414 16373 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:39.245095 15996 ts_manager.cc:194] Registered new tserver with Master: e0c7e2cdf2eb457db9a026353414b0e3 (127.2.223.130:42039)
I20260807 11:19:39.245885 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:35225
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:19:39.298669 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:19:39.555125 16218 heartbeater.cc:503] Master 127.2.223.190:33615 was elected leader, sending a full tablet report...
W20260807 11:19:39.648262 16380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:39.648563 16380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:39.648752 16380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:39.656519 16380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:39.656661 16380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:39.656862 16380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:19:39.667390 16380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:39.668668 16380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:39.669961 16380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:39.684756 16385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:39.684725 16386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:39.684916 16388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:39.685983 16380 server_base.cc:1034] running on GCE node
I20260807 11:19:39.686728 16380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:39.687740 16380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:39.689054 16380 hybrid_clock.cc:648] HybridClock initialized: now 1786101579688917 us; error 53 us; skew 500 ppm
I20260807 11:19:39.689491 16380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:39.694116 16380 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:19:39.696173 16380 webserver.cc:460] Webserver started at http://127.2.223.131:39141/ using document root <none> and password file <none>
I20260807 11:19:39.696807 16380 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:39.696915 16380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:39.697238 16380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:39.715690 16380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "ee001397238a4783a68e7361d23ab443"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "f099b6dd64c3c06401e43348e614746c"
server_key_iv: "473b59b2f28d5d7103feb912aa42577c"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.717060 16380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "ee001397238a4783a68e7361d23ab443"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "f099b6dd64c3c06401e43348e614746c"
server_key_iv: "473b59b2f28d5d7103feb912aa42577c"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.725397 16380 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260807 11:19:39.744695 16395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:39.746680 16380 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260807 11:19:39.746933 16380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "ee001397238a4783a68e7361d23ab443"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "f099b6dd64c3c06401e43348e614746c"
server_key_iv: "473b59b2f28d5d7103feb912aa42577c"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:39.747292 16380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:39.772270 16380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:39.778194 16380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:39.778654 16380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:39.780071 16380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:39.782194 16380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:39.782373 16380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:39.782565 16380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:39.782747 16380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 11:19:39.875037 16380 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:43943
I20260807 11:19:39.875159 16517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:43943 every 8 connection(s)
I20260807 11:19:39.876610 16380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:19:39.883100 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16380
I20260807 11:19:39.883354 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:19:39.884238 2942 external_mini_cluster.cc:1428] Setting key dab39cf74ee9ea4e2bce1962cc3e5e46
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:39.906566 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:39.881023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:49715 (local address 127.2.223.190:33615)
0807 11:19:39.881773 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:39.881777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:39.883727 (+ 1950us) server_negotiation.cc:407] Connection header received
0807 11:19:39.887473 (+ 3746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:39.887477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:39.887575 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:39.887666 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:39.889780 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.891004 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.893500 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.894174 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.894297 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:39.898702 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:39.898707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:39.898710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:39.898769 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:39.903012 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.903800 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.903804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.903806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.903976 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.904572 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.904576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.904578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.904948 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:39.905091 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:39.905836 (+ 745us) server_negotiation.cc:299] Negotiation successful
0807 11:19:39.906066 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":493,"threads_started":1}
I20260807 11:19:39.907388 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:39.881996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.131:49715)
0807 11:19:39.883528 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:39.883593 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:39.887146 (+ 3553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:39.887971 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:39.887984 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:39.888741 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:39.889615 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:39.889636 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.891287 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.891291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:39.893362 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:39.893376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.894573 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.894582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:39.894863 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:39.895652 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:39.895674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:39.898400 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:39.903172 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:39.903179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:39.903192 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:39.903695 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:39.904129 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:39.904133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:39.904135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:39.904458 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:39.905082 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:39.905087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:39.905419 (+ 332us) client_negotiation.cc:769] Sending connection context
0807 11:19:39.905648 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 11:19:39.906081 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":626,"threads_started":1}
I20260807 11:19:39.909164 16518 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:39.909509 16518 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:39.910398 16518 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:39.912276 15996 ts_manager.cc:194] Registered new tserver with Master: ee001397238a4783a68e7361d23ab443 (127.2.223.131:43943)
I20260807 11:19:39.913429 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:49715
I20260807 11:19:39.914439 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:19:39.916515 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16079
Aug 07 11:19:39 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:19:39.940412 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:39.917616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.2.223.129:44443)
0807 11:19:39.919056 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:39.919063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:39.919096 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:19:39.919162 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:39.919169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:39.919361 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:39.919508 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:39.921641 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.923215 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.925319 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:39.926099 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:39.926222 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:39.931971 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:39.931984 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:39.931998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:39.932055 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:39.936882 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.937946 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.937953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.937955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.938231 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:39.938752 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:39.938756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:39.938757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:39.939135 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:39.939501 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:39.939649 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 11:19:39.939890 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":928,"threads_started":1}
I20260807 11:19:39.941871 16212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:40.028966 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16079
I20260807 11:19:40.050218 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16225
Aug 07 11:19:40 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:19:40.068074 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:40.051383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.2.223.130:42039)
0807 11:19:40.051956 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:40.051963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:40.051989 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:19:40.052059 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:40.052065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:40.052284 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:40.052409 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:40.054027 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:40.055246 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:40.057170 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:40.057766 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:40.057942 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:40.062646 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:40.062657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:40.062682 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:40.062717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:40.065506 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:40.066199 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:40.066206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:40.066208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:40.066338 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:40.066731 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:40.066734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:40.066736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:40.067049 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:40.067279 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:40.067487 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 11:19:40.067689 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":318,"threads_started":1}
I20260807 11:19:40.069396 16367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:40.151968 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16225
I20260807 11:19:40.171514 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16380
Aug 07 11:19:40 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:19:40.189255 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:40.172738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.2.223.131:43943)
0807 11:19:40.173455 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:40.173463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:40.173492 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:19:40.173553 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:40.173559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:40.173778 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:40.173906 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:40.175438 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:40.176466 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:40.177868 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:40.178419 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:40.178604 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:40.183162 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:40.183173 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:40.183188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:40.183228 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:40.186237 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:40.187046 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:40.187054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:40.187056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:40.187197 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:40.187571 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:40.187574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:40.187576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:40.187910 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:40.188168 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:40.188545 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 11:19:40.188866 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":406,"threads_started":1}
I20260807 11:19:40.190840 16512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:40.277164 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16380
I20260807 11:19:40.297108 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 15962
I20260807 11:19:40.298384 16027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:40.407908 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 15962
I20260807 11:19:40.437209 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33615
--webserver_interface=127.2.223.190
--webserver_port=38947
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33615
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:40.770397 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:40.770697 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:40.770872 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:40.777802 16530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:40.777897 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:40.777989 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:40.778095 16530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:40.778187 16530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:40.787845 16530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33615
--ranger_config_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=38947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:40.789086 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:40.790445 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:40.802050 16536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:40.802117 16537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:40.802176 16539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:40.803144 16530 server_base.cc:1034] running on GCE node
I20260807 11:19:40.803943 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:40.805374 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:40.806594 16530 hybrid_clock.cc:648] HybridClock initialized: now 1786101580806555 us; error 52 us; skew 500 ppm
I20260807 11:19:40.806953 16530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:40 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:40.810236 16530 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:40.811882 16530 webserver.cc:460] Webserver started at http://127.2.223.190:38947/ using document root <none> and password file <none>
I20260807 11:19:40.812543 16530 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:40.812680 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:19:40 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101580, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:40.891855 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 11:19:40.895066 16547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:40.896602 16530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:19:40.896835 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "325be22e4d2c483a8825aa6019d0bbc1"
format_stamp: "Formatted at 2026-08-07 11:19:36 on dist-test-slave-5l3k"
server_key: "04afe5b1daffd11c8243b7417592b8c9"
server_key_iv: "7decaaa16fc1365e8157d4ad122ebf88"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:40.897864 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:40.912653 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:40.916481 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:40.916779 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:40.935590 16530 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33615
I20260807 11:19:40.935629 16600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33615 every 8 connection(s)
I20260807 11:19:40.937103 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:40.937896 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16530
I20260807 11:19:40.938973 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:44443
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:33615
--webserver_port=35729
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:19:40.944510 16601 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:19:40.947795 16601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Bootstrap starting.
I20260807 11:19:40.971156 16601 log.cc:824] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:40.980399 16601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:19:40.980947 16601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Bootstrap complete.
I20260807 11:19:40.990129 16601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:40.991039 16601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325be22e4d2c483a8825aa6019d0bbc1, State: Initialized, Role: FOLLOWER
I20260807 11:19:40.991698 16601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:40.991932 16601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:19:40.992043 16601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:19:40.992201 16601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:19:40.996263 16601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:40.996829 16601 leader_election.cc:302] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325be22e4d2c483a8825aa6019d0bbc1; no voters:
I20260807 11:19:40.997478 16601 leader_election.cc:288] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:19:40.997635 16605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:19:40.998745 16605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [term 2 LEADER]: Becoming Leader. State: Replica: 325be22e4d2c483a8825aa6019d0bbc1, State: Running, Role: LEADER
I20260807 11:19:40.999406 16605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } }
I20260807 11:19:40.999778 16601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:41.002466 16606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "325be22e4d2c483a8825aa6019d0bbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } } }
I20260807 11:19:41.003094 16606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:41.002663 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325be22e4d2c483a8825aa6019d0bbc1. Latest consensus state: current_term: 2 leader_uuid: "325be22e4d2c483a8825aa6019d0bbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325be22e4d2c483a8825aa6019d0bbc1" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33615 } } }
I20260807 11:19:41.004698 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:41.013646 16612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:41.019925 16612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:41.020785 16601 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 11:19:41.021457 16612 catalog_manager.cc:1259] Loaded cluster ID: 07548abe66674c3bb80775aa721837bc
I20260807 11:19:41.021620 16612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:41.025601 16612 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:41.027060 16612 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Loaded TSK: 0
I20260807 11:19:41.028581 16612 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:19:41.421602 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:41.422109 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:41.422351 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:41.433480 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:41.433650 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:41.433848 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:19:41.447265 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=35729
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:41.449250 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:41.451165 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:41.493312 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:41.493305 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:41.494953 16603 server_base.cc:1034] running on GCE node
W20260807 11:19:41.495666 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:41.496592 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:41.506856 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:41.508153 16603 hybrid_clock.cc:648] HybridClock initialized: now 1786101581508075 us; error 77 us; skew 500 ppm
I20260807 11:19:41.508528 16603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:41 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:41.512180 16603 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:19:41.515821 16603 webserver.cc:460] Webserver started at http://127.2.223.129:35729/ using document root <none> and password file <none>
I20260807 11:19:41.516566 16603 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:41.516742 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:19:41 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:41.575294 16603 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.009s sys 0.001s
I20260807 11:19:41.588487 16651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:41.590610 16603 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.010s
I20260807 11:19:41.590888 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc"
format_stamp: "Formatted at 2026-08-07 11:19:38 on dist-test-slave-5l3k"
server_key: "16fa76590a725fea93aa02ceec62bf0b"
server_key_iv: "6eb4a803e0b892365ad0536061b05ea7"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:41.592433 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:41.643973 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:41.655711 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:41.657778 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:41.666062 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:41.674250 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:41.674436 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:41.674670 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:41.674816 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:41.780164 16603 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:44443
I20260807 11:19:41.780257 16772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:44443 every 8 connection(s)
I20260807 11:19:41.782222 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:19:41.784397 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16603
I20260807 11:19:41.785269 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:42039
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:33615
--webserver_port=40541
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:41 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101581, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:41.839413 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:41.787639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38865 (local address 127.2.223.190:33615)
0807 11:19:41.802839 (+ 15200us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:41.802860 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:41.802919 (+ 59us) server_negotiation.cc:407] Connection header received
0807 11:19:41.807383 (+ 4464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:41.807407 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:41.807935 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:41.808381 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:41.811290 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:41.813212 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:41.816680 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:41.817661 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:41.817830 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:41.826184 (+ 8354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:41.826195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:41.826215 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:41.826269 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:41.830819 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:41.831837 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:41.831846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:41.831855 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:41.832070 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:41.833839 (+ 1769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:41.833857 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:41.833860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:41.834587 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:41.834777 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:41.837628 (+ 2851us) server_negotiation.cc:299] Negotiation successful
0807 11:19:41.838076 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14893,"thread_start_us":15179,"threads_started":1}
I20260807 11:19:41.848469 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:41.786508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.129:38865)
0807 11:19:41.798280 (+ 11772us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:41.798352 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:41.804180 (+ 5828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:41.809392 (+ 5212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:41.809406 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:41.810231 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:41.811055 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:41.811081 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:41.814437 (+ 3356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:41.814444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:41.816459 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:41.816506 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:41.818288 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:41.818296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:41.818590 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:41.819422 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:41.819452 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:41.824969 (+ 5517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:41.831004 (+ 6035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:41.831014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:41.831044 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:41.831659 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:41.833420 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:41.833426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:41.833430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:41.833695 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:41.834860 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:41.834868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:41.835276 (+ 408us) client_negotiation.cc:769] Sending connection context
0807 11:19:41.835577 (+ 301us) client_negotiation.cc:240] Negotiation successful
0807 11:19:41.836171 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11400,"thread_start_us":11159,"threads_started":1}
I20260807 11:19:41.852598 16773 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:41.852919 16773 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:41.853780 16773 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
W20260807 11:19:41.857466 16773 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:33615 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:19:41.857702 16773 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:41.857985 16773 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:41.860306 16773 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:41.860612 16773 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
W20260807 11:19:41.861668 16773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:19:42.134351 16784 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 11:19:42.249205 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:42.249642 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:42.249811 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:42.259457 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:42.259593 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:42.259783 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:19:42.270058 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:42039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=40541
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:42.271402 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:42.272933 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:42.290714 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:42.291301 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:42.299296 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:42.301137 16777 server_base.cc:1034] running on GCE node
I20260807 11:19:42.301889 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:42.303182 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:42.304481 16777 hybrid_clock.cc:648] HybridClock initialized: now 1786101582304413 us; error 54 us; skew 500 ppm
I20260807 11:19:42.304813 16777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:42 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:42.309080 16777 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:19:42.310916 16777 webserver.cc:460] Webserver started at http://127.2.223.130:40541/ using document root <none> and password file <none>
I20260807 11:19:42.311736 16777 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:42.311906 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:19:42 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:42.343142 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:19:42.359854 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:42.362450 16777 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.008s sys 0.007s
I20260807 11:19:42.362737 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "e0c7e2cdf2eb457db9a026353414b0e3"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "9e04e3aa9f73407cd07893fa7bc6282c"
server_key_iv: "2d3b2d25e89a44cefdbe429a1a537685"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:42.364187 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:42.415782 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:42.421247 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:42.421571 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:42.423203 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:42.424952 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:42.425071 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:42.425180 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:42.425272 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:19:42 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:42.504724 16777 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42039
I20260807 11:19:42.504777 16928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42039 every 8 connection(s)
I20260807 11:19:42.506793 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:19:42.517076 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16777
I20260807 11:19:42.517809 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.223.148:50527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:43943
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:33615
--webserver_port=39141
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:42 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:42.532817 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:42.510015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:59625 (local address 127.2.223.190:33615)
0807 11:19:42.513142 (+ 3127us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:42.513148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:42.513172 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:19:42.514978 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:42.514985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:42.515083 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:42.515200 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:42.516634 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:42.517865 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:42.520255 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:42.521013 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:42.521176 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:42.525477 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:42.525484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:42.525488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:42.525533 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:42.529155 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:42.530201 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:42.530207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:42.530211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:42.530369 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:42.530982 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:42.530987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:42.530991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:42.531346 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:42.531519 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:42.532206 (+ 687us) server_negotiation.cc:299] Negotiation successful
0807 11:19:42.532439 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2991,"thread_start_us":2824,"threads_started":1}
I20260807 11:19:42.533819 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:42.510438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.130:59625)
0807 11:19:42.512538 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:42.512610 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:42.514719 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:42.515431 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:42.515440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:42.515892 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:42.516465 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:42.516482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:42.518054 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:42.518060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:42.520074 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:42.520098 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:42.521256 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:42.521263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:42.521537 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:42.522310 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:42.522339 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:42.525227 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:42.529338 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:42.529350 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:42.529369 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:42.530041 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:42.530520 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:42.530526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:42.530530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:42.530812 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:42.531483 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:42.531489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:42.531834 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 11:19:42.532102 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 11:19:42.532633 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"thread_start_us":1397,"threads_started":1}
I20260807 11:19:42.535655 16929 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:42.535961 16929 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:42.536837 16929 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:42.538931 16564 ts_manager.cc:194] Registered new tserver with Master: e0c7e2cdf2eb457db9a026353414b0e3 (127.2.223.130:42039)
I20260807 11:19:42.540557 16564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:59625
I20260807 11:19:42.861858 16773 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:42.862336 16773 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:42.863734 16564 ts_manager.cc:194] Registered new tserver with Master: ff33342e1e4340cf8b35fb4ad3a9d5bc (127.2.223.129:44443)
I20260807 11:19:42.864915 16564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:38865
Aug 07 11:19:42 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
W20260807 11:19:42.992865 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:42.993322 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:42.993538 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:43.001868 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:43.002015 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:43.002202 16933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:19:43.012606 16933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.223.148:50527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:43943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=39141
--tserver_master_addrs=127.2.223.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:43.014063 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:43.015477 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:43.029632 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:43.031442 16940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:43.036537 16942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:43.037209 16933 server_base.cc:1034] running on GCE node
I20260807 11:19:43.037834 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:43.038895 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:43.040135 16933 hybrid_clock.cc:648] HybridClock initialized: now 1786101583040063 us; error 69 us; skew 500 ppm
I20260807 11:19:43.040442 16933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:43 dist-test-slave-5l3k krb5kdc[15492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101583, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:43.043732 16933 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:19:43.045253 16933 webserver.cc:460] Webserver started at http://127.2.223.131:39141/ using document root <none> and password file <none>
I20260807 11:19:43.045902 16933 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:43.046020 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:19:43 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101583, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:43.072292 16933 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 11:19:43.075857 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:43.077702 16933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260807 11:19:43.077884 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "ee001397238a4783a68e7361d23ab443"
format_stamp: "Formatted at 2026-08-07 11:19:39 on dist-test-slave-5l3k"
server_key: "f099b6dd64c3c06401e43348e614746c"
server_key_iv: "473b59b2f28d5d7103feb912aa42577c"
server_key_version: "kuduclusterkey@0"
I20260807 11:19:43.079197 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:43.102785 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:43.108237 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:43.108556 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:43.109838 16933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:43.111662 16933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:43.111820 16933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 11:19:43.111948 16933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:43.112025 16933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:43.162829 16933 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:43943
I20260807 11:19:43.162921 17073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:43943 every 8 connection(s)
I20260807 11:19:43.164793 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:19:43.168147 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 16933
Aug 07 11:19:43 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101583, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:43.209165 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.170811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.2.223.190:33615)
0807 11:19:43.184896 (+ 14085us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.184902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.184922 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:19:43.184972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.184977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.185058 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.185752 (+ 694us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:43.190925 (+ 5173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.193095 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.195167 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.195836 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.195928 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.200311 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:43.200319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:43.200324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:43.200377 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:43.205739 (+ 5362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.206756 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.206761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.206765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.206922 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.207402 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.207407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.207410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.207699 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:43.207925 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.208508 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.208778 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":13948,"thread_start_us":12253,"threads_started":1}
I20260807 11:19:43.210011 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.170592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:36129 (local address 127.2.223.190:33615)
0807 11:19:43.182922 (+ 12330us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.182929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.182956 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:19:43.183590 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.183598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.183695 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.183773 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:43.185931 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.187059 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.189401 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.190071 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.190166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.196273 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:43.196280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:43.196285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:43.196336 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:43.204077 (+ 7741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.205989 (+ 1912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.205995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.205999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.206166 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.206823 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.206828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.206831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.207213 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:43.207341 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.208485 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.209625 (+ 1140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":870,"server-negotiator.queue_time_us":12192,"thread_start_us":12380,"threads_started":1}
I20260807 11:19:43.210964 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.169854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:33615 (local address 127.2.223.131:36129)
0807 11:19:43.179333 (+ 9479us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.179397 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.183235 (+ 3838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.184320 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.184333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.184920 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:43.185722 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.185746 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.187268 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.187273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.189233 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.189249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.190544 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.190551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.190835 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.191657 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:43.191686 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:43.195981 (+ 4295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:43.205088 (+ 9107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:43.205099 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:43.205120 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:43.205832 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:43.206332 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:43.206337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:43.206341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:43.206656 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:43.207472 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:43.207480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:43.207915 (+ 435us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.208649 (+ 734us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.209272 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8980,"spinlock_wait_cycles":26368,"thread_start_us":8748,"threads_started":1}
I20260807 11:19:43.212805 17074 heartbeater.cc:347] Connected to a master server at 127.2.223.190:33615
I20260807 11:19:43.213156 17074 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:43.214190 17074 heartbeater.cc:511] Master 127.2.223.190:33615 requested a full tablet report, sending...
I20260807 11:19:43.215876 16564 ts_manager.cc:194] Registered new tserver with Master: ee001397238a4783a68e7361d23ab443 (127.2.223.131:43943)
I20260807 11:19:43.216872 16564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:36129
I20260807 11:19:43.230044 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:19:43 dist-test-slave-5l3k krb5kdc[15492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101582, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for HTTP/127.2.223.148@KRBTEST.COM
I20260807 11:19:43.270746 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.250708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.2.223.190:33615)
0807 11:19:43.250910 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.250916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.251514 (+ 598us) server_negotiation.cc:407] Connection header received
0807 11:19:43.251847 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.251854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.251956 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.252037 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:43.254330 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.255453 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.257591 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.258297 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.258398 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.263403 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:43.263410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:43.263415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:43.263459 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:43.267149 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.268049 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.268055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.268058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.268220 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.268733 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.268738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.268741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.269154 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:43.269384 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.270038 (+ 654us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.270299 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:19:43.278816 16564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:19:43.290455 16564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:19:43.367892 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.352340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44443 (local address 127.0.0.1:55212)
0807 11:19:43.356672 (+ 4332us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.356696 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.356838 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.359625 (+ 2787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.359629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.359659 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:43.360169 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.360187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.362511 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.362516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.364800 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.364819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.366829 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.366835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.367069 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.367135 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.367251 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.367447 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4169,"mutex_wait_us":35,"thread_start_us":4589,"threads_started":1}
I20260807 11:19:43.369647 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.352659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.2.223.129:44443)
0807 11:19:43.358981 (+ 6322us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.358989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.359018 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:19:43.359117 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.359125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.359367 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.360376 (+ 1009us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:43.360650 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.362316 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.365018 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.366573 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.368732 (+ 2159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.369038 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.369117 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.369286 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6158,"thread_start_us":6494,"threads_started":1}
I20260807 11:19:43.372434 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.326018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:43943 (local address 127.0.0.1:46906)
0807 11:19:43.353343 (+ 27325us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.353362 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.353483 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.354631 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.354639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.354671 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:43.355109 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.355122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.360065 (+ 4943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.360070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.364877 (+ 4807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.364892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.371639 (+ 6747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.371645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.371817 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.371850 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.371921 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.372066 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25800,"mutex_wait_us":4,"thread_start_us":25098,"threads_started":1}
I20260807 11:19:43.373920 16703 tablet_service.cc:1467] Processing CreateTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 (DEFAULT_TABLE table=test-table [id=128150a6b36d49d3bdaed764f3779d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:43.375486 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.323903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:42039 (local address 127.0.0.1:55278)
0807 11:19:43.351534 (+ 27631us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.351962 (+ 428us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.352188 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.359354 (+ 7166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.359359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.359384 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:43.359831 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.359844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.367884 (+ 8040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.367889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.371408 (+ 3519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.371424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.374617 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.374623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.374792 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.374824 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.374913 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.375121 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27336,"mutex_wait_us":1,"thread_start_us":28599,"threads_started":1}
I20260807 11:19:43.375954 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991e509c4ec54d449a47b5ca29d6ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:43.376343 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.326424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.2.223.131:43943)
0807 11:19:43.350700 (+ 24276us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.350711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.353621 (+ 2910us) server_negotiation.cc:407] Connection header received
0807 11:19:43.353739 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.353747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.354039 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.354258 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:43.355334 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.356973 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.365079 (+ 8106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.366595 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.366717 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.366980 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.375796 (+ 8816us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.375999 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24038,"thread_start_us":24309,"threads_started":1}
I20260807 11:19:43.378115 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.323948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.2.223.130:42039)
0807 11:19:43.358510 (+ 34562us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.358521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.358551 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:19:43.358643 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.358650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.358897 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.359104 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:43.361926 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.367696 (+ 5770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.372654 (+ 4958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.374146 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.374268 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.374444 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.377575 (+ 3131us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.377777 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34352,"thread_start_us":37733,"threads_started":1}
I20260807 11:19:43.380261 17004 tablet_service.cc:1467] Processing CreateTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 (DEFAULT_TABLE table=test-table [id=128150a6b36d49d3bdaed764f3779d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:43.381778 17004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991e509c4ec54d449a47b5ca29d6ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:43.381685 16857 tablet_service.cc:1467] Processing CreateTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 (DEFAULT_TABLE table=test-table [id=128150a6b36d49d3bdaed764f3779d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:43.383070 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991e509c4ec54d449a47b5ca29d6ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:43.411300 17097 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Bootstrap starting.
I20260807 11:19:43.412096 17098 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Bootstrap starting.
I20260807 11:19:43.416185 17097 tablet_bootstrap.cc:654] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:43.417806 17097 log.cc:824] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:43.419015 17098 tablet_bootstrap.cc:654] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:43.421177 17098 log.cc:824] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:43.422089 17099 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Bootstrap starting.
I20260807 11:19:43.430344 17097 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: No bootstrap required, opened a new log
I20260807 11:19:43.430876 17097 ts_tablet_manager.cc:1397] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20260807 11:19:43.437918 17099 tablet_bootstrap.cc:654] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:43.447310 17099 log.cc:824] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:43.452363 17097 raft_consensus.cc:348] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.454116 17097 raft_consensus.cc:374] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:43.454393 17097 raft_consensus.cc:729] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee001397238a4783a68e7361d23ab443, State: Initialized, Role: FOLLOWER
I20260807 11:19:43.457015 17097 consensus_queue.cc:260] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.458146 17098 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: No bootstrap required, opened a new log
I20260807 11:19:43.458541 17098 ts_tablet_manager.cc:1397] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Time spent bootstrapping tablet: real 0.047s user 0.022s sys 0.018s
I20260807 11:19:43.459525 17099 tablet_bootstrap.cc:492] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: No bootstrap required, opened a new log
I20260807 11:19:43.459925 17099 ts_tablet_manager.cc:1397] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.005s
I20260807 11:19:43.473771 17098 raft_consensus.cc:348] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.474445 17098 raft_consensus.cc:374] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:43.474655 17098 raft_consensus.cc:729] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c7e2cdf2eb457db9a026353414b0e3, State: Initialized, Role: FOLLOWER
I20260807 11:19:43.475734 17074 heartbeater.cc:503] Master 127.2.223.190:33615 was elected leader, sending a full tablet report...
I20260807 11:19:43.476549 17097 ts_tablet_manager.cc:1428] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Time spent starting tablet: real 0.045s user 0.024s sys 0.010s
I20260807 11:19:43.476650 17098 consensus_queue.cc:260] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.480397 17099 raft_consensus.cc:348] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.481045 17099 raft_consensus.cc:374] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:43.481243 17099 raft_consensus.cc:729] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff33342e1e4340cf8b35fb4ad3a9d5bc, State: Initialized, Role: FOLLOWER
I20260807 11:19:43.482214 17099 consensus_queue.cc:260] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.485891 16773 heartbeater.cc:503] Master 127.2.223.190:33615 was elected leader, sending a full tablet report...
I20260807 11:19:43.486202 17099 ts_tablet_manager.cc:1428] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Time spent starting tablet: real 0.026s user 0.016s sys 0.007s
I20260807 11:19:43.486239 17098 ts_tablet_manager.cc:1428] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Time spent starting tablet: real 0.028s user 0.013s sys 0.010s
I20260807 11:19:43.486915 16929 heartbeater.cc:503] Master 127.2.223.190:33615 was elected leader, sending a full tablet report...
W20260807 11:19:43.510711 16930 tablet.cc:2369] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:19:43.558229 16774 tablet.cc:2369] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:43.639372 17104 raft_consensus.cc:482] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:43.639878 17104 raft_consensus.cc:504] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.641965 17104 leader_election.cc:288] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee001397238a4783a68e7361d23ab443 (127.2.223.131:43943), ff33342e1e4340cf8b35fb4ad3a9d5bc (127.2.223.129:44443)
I20260807 11:19:43.650389 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.642849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:60221 (local address 127.2.223.131:43943)
0807 11:19:43.643148 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.643155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.643605 (+ 450us) server_negotiation.cc:407] Connection header received
0807 11:19:43.643865 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.643871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.643964 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.644061 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:43.644985 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.646182 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.648430 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.649312 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.649507 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.649596 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.649913 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.650065 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260807 11:19:43.650673 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.642641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:43943 (local address 127.2.223.130:60221)
0807 11:19:43.643452 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.643469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.643692 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.644176 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.644181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.644226 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:43.644829 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.644846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.646330 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.646348 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.648258 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.648271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.649574 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.649581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.649753 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.649778 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.649885 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.650070 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":52,"thread_start_us":480,"threads_started":1}
I20260807 11:19:43.650909 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.643803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:33039 (local address 127.2.223.129:44443)
0807 11:19:43.644064 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.644072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.644681 (+ 609us) server_negotiation.cc:407] Connection header received
0807 11:19:43.644911 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.644920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.645037 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.645110 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:43.645975 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.646817 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.648884 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.649876 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.650072 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.650156 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.650467 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.650627 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:19:43.651170 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.643586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:44443 (local address 127.2.223.130:33039)
0807 11:19:43.644531 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:43.644569 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:43.644766 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:43.645151 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:43.645156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:43.645185 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:43.645795 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.645807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.646954 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.646960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.648711 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:43.648723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.650155 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.650163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:43.650310 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.650332 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 11:19:43.650439 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:19:43.650587 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":2,"thread_start_us":502,"threads_started":1}
I20260807 11:19:43.652103 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991e509c4ec54d449a47b5ca29d6ae58" candidate_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee001397238a4783a68e7361d23ab443" is_pre_election: true
I20260807 11:19:43.652241 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991e509c4ec54d449a47b5ca29d6ae58" candidate_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" is_pre_election: true
I20260807 11:19:43.652734 17026 raft_consensus.cc:2454] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c7e2cdf2eb457db9a026353414b0e3 in term 0.
I20260807 11:19:43.652904 16725 raft_consensus.cc:2454] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c7e2cdf2eb457db9a026353414b0e3 in term 0.
I20260807 11:19:43.653473 16807 leader_election.cc:302] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0c7e2cdf2eb457db9a026353414b0e3, ee001397238a4783a68e7361d23ab443; no voters:
I20260807 11:19:43.653887 17104 raft_consensus.cc:2781] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:19:43.654094 17104 raft_consensus.cc:482] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:19:43.654242 17104 raft_consensus.cc:3037] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:43.657035 17104 raft_consensus.cc:504] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.657917 17104 leader_election.cc:288] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [CANDIDATE]: Term 1 election: Requested vote from peers ee001397238a4783a68e7361d23ab443 (127.2.223.131:43943), ff33342e1e4340cf8b35fb4ad3a9d5bc (127.2.223.129:44443)
I20260807 11:19:43.658394 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991e509c4ec54d449a47b5ca29d6ae58" candidate_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee001397238a4783a68e7361d23ab443"
I20260807 11:19:43.658573 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991e509c4ec54d449a47b5ca29d6ae58" candidate_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc"
I20260807 11:19:43.658681 17026 raft_consensus.cc:3037] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:43.658833 16725 raft_consensus.cc:3037] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:43.661636 17026 raft_consensus.cc:2454] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c7e2cdf2eb457db9a026353414b0e3 in term 1.
I20260807 11:19:43.661861 16725 raft_consensus.cc:2454] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c7e2cdf2eb457db9a026353414b0e3 in term 1.
I20260807 11:19:43.662159 16807 leader_election.cc:302] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0c7e2cdf2eb457db9a026353414b0e3, ee001397238a4783a68e7361d23ab443; no voters:
I20260807 11:19:43.662616 17104 raft_consensus.cc:2781] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:43.663424 17104 raft_consensus.cc:686] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 1 LEADER]: Becoming Leader. State: Replica: e0c7e2cdf2eb457db9a026353414b0e3, State: Running, Role: LEADER
I20260807 11:19:43.664247 17104 consensus_queue.cc:237] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } }
I20260807 11:19:43.669301 16564 catalog_manager.cc:5495] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0c7e2cdf2eb457db9a026353414b0e3 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c7e2cdf2eb457db9a026353414b0e3" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 } health_report { overall_health: UNKNOWN } } }
W20260807 11:19:43.674032 17075 tablet.cc:2369] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:43.722246 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.714931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.2.223.130:42039)
0807 11:19:43.715179 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.715185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.715486 (+ 301us) server_negotiation.cc:407] Connection header received
0807 11:19:43.715744 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.715754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.715875 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.716029 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:19:43.716611 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.717612 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.719550 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.720114 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.720250 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.720917 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:19:43.721499 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:19:43.721692 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.721775 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.721863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:19:43.732018 17026 raft_consensus.cc:1261] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 1 FOLLOWER]: Refusing update from remote peer e0c7e2cdf2eb457db9a026353414b0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:43.732033 16725 raft_consensus.cc:1261] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 1 FOLLOWER]: Refusing update from remote peer e0c7e2cdf2eb457db9a026353414b0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:43.733170 17116 consensus_queue.cc:1035] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff33342e1e4340cf8b35fb4ad3a9d5bc" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:43.733769 17104 consensus_queue.cc:1035] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee001397238a4783a68e7361d23ab443" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:43.748513 17119 mvcc.cc:205] Tried to move back new op lower bound from 7315872086954577920 to 7315872086692839424. Current Snapshot: MvccSnapshot[applied={T|T < 7315872086954577920}]
I20260807 11:19:43.769168 16564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:43.769693 16564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:43.774683 16564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325be22e4d2c483a8825aa6019d0bbc1: Sending DeleteTablet for 3 replicas of tablet 991e509c4ec54d449a47b5ca29d6ae58
I20260807 11:19:43.776180 16857 tablet_service.cc:1514] Processing DeleteTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:43 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:55278
I20260807 11:19:43.776513 17004 tablet_service.cc:1514] Processing DeleteTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:43 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:46906
I20260807 11:19:43.776979 16703 tablet_service.cc:1514] Processing DeleteTablet for tablet 991e509c4ec54d449a47b5ca29d6ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:43 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:55212
I20260807 11:19:43.777714 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16603
I20260807 11:19:43.792433 17132 tablet_replica.cc:330] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: stopping tablet replica
I20260807 11:19:43.795533 17132 raft_consensus.cc:2229] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:43.796535 17133 tablet_replica.cc:330] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: stopping tablet replica
I20260807 11:19:43.797231 17133 raft_consensus.cc:2229] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:19:43.799172 17131 tablet_replica.cc:330] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: stopping tablet replica
I20260807 11:19:43.799703 17131 raft_consensus.cc:2229] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:43.800942 17133 raft_consensus.cc:2258] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:43.801262 17132 raft_consensus.cc:2258] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:43.804188 17133 ts_tablet_manager.cc:1905] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:43.804903 17131 raft_consensus.cc:2258] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:43.807936 17131 ts_tablet_manager.cc:1905] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:43.810105 17132 ts_tablet_manager.cc:1905] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:43.818382 17131 ts_tablet_manager.cc:1918] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:43.818661 17131 log.cc:1196] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/991e509c4ec54d449a47b5ca29d6ae58
I20260807 11:19:43.819734 17131 ts_tablet_manager.cc:1939] T 991e509c4ec54d449a47b5ca29d6ae58 P ee001397238a4783a68e7361d23ab443: Deleting consensus metadata
I20260807 11:19:43.820807 17133 ts_tablet_manager.cc:1918] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:43.821120 17133 log.cc:1196] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/991e509c4ec54d449a47b5ca29d6ae58
I20260807 11:19:43.822094 17133 ts_tablet_manager.cc:1939] T 991e509c4ec54d449a47b5ca29d6ae58 P e0c7e2cdf2eb457db9a026353414b0e3: Deleting consensus metadata
I20260807 11:19:43.822131 16548 catalog_manager.cc:4841] TS ee001397238a4783a68e7361d23ab443 (127.2.223.131:43943): tablet 991e509c4ec54d449a47b5ca29d6ae58 (table test-table [id=128150a6b36d49d3bdaed764f3779d6f]) successfully deleted
I20260807 11:19:43.825038 17132 ts_tablet_manager.cc:1918] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:43.825160 16551 catalog_manager.cc:4841] TS e0c7e2cdf2eb457db9a026353414b0e3 (127.2.223.130:42039): tablet 991e509c4ec54d449a47b5ca29d6ae58 (table test-table [id=128150a6b36d49d3bdaed764f3779d6f]) successfully deleted
I20260807 11:19:43.825408 17132 log.cc:1196] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/991e509c4ec54d449a47b5ca29d6ae58
I20260807 11:19:43.826589 17132 ts_tablet_manager.cc:1939] T 991e509c4ec54d449a47b5ca29d6ae58 P ff33342e1e4340cf8b35fb4ad3a9d5bc: Deleting consensus metadata
I20260807 11:19:43.829226 16548 catalog_manager.cc:4841] TS ff33342e1e4340cf8b35fb4ad3a9d5bc (127.2.223.129:44443): tablet 991e509c4ec54d449a47b5ca29d6ae58 (table test-table [id=128150a6b36d49d3bdaed764f3779d6f]) successfully deleted
I20260807 11:19:43.829981 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.778731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.2.223.129:44443)
0807 11:19:43.778950 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.778960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.787065 (+ 8105us) server_negotiation.cc:407] Connection header received
0807 11:19:43.788128 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.788136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.788259 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.788347 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:43.796841 (+ 8494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.798876 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.803392 (+ 4516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.809759 (+ 6367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.809906 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.815546 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:43.815563 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:43.815593 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:43.815715 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:43.825288 (+ 9573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.826436 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.826455 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.826457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.826837 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:43.827373 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:43.827379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:43.827383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:43.827958 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:43.828541 (+ 583us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:43.828898 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 11:19:43.829219 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:19:43.830935 16768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:43.963737 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16603
I20260807 11:19:43.986665 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16777
I20260807 11:19:44.003043 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:43.987544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.2.223.130:42039)
0807 11:19:43.987751 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:43.987762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:43.987785 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:19:43.987974 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:43.987980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:43.988062 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:43.988201 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:43.989435 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.990886 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.992296 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:43.992964 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:43.993093 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:43.995929 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:43.995947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:43.995969 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:43.996008 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:44.000006 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:44.000738 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:44.000752 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:44.000756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:44.000926 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:44.001362 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:44.001368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:44.001371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:44.001817 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:44.002184 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:44.002388 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 11:19:44.002628 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:19:44.003777 16924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:44.119429 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16777
I20260807 11:19:44.141845 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16933
I20260807 11:19:44.155939 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:44.142750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.2.223.131:43943)
0807 11:19:44.142933 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:44.142938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:44.142957 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:19:44.143166 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:44.143171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:44.143250 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:44.143343 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:44.144888 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:44.145607 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:44.147025 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:44.147415 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:44.147509 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:44.149961 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:44.149978 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:44.149996 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:44.150031 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:44.152911 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:44.153583 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:44.153591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:44.153593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:44.153766 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:44.154202 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:44.154206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:44.154208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:44.154907 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:44.155135 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:44.155440 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 11:19:44.155627 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 11:19:44.156647 17068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:44.245767 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16933
I20260807 11:19:44.266010 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 16530
I20260807 11:19:44.267343 16595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:44.385592 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 16530
I20260807 11:19:44.416132 2942 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 11:19:44.761049 2942 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 11:19:44.761281 2942 mini_ranger.cc:66] Stopping Ranger...
I20260807 11:19:45.107754 2942 mini_ranger.cc:68] Stopped Ranger
2026-08-07 11:19:45.108 UTC [15526] LOG: received smart shutdown request
2026-08-07 11:19:45.111 UTC [15526] LOG: background worker "logical replication launcher" (PID 15536) exited with exit code 1
2026-08-07 11:19:45.111 UTC [15530] LOG: shutting down
2026-08-07 11:19:45.290 UTC [15526] LOG: database system is shut down
2026-08-07T11:19:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56474 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:19:45 dist-test-slave-5l3k krb5kdc[17148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:19:45 dist-test-slave-5l3k krb5kdc[17148](info): set up 2 sockets
Aug 07 11:19:45 dist-test-slave-5l3k krb5kdc[17148](info): commencing operation
krb5kdc: starting...
W20260807 11:19:46.595703 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:19:46 dist-test-slave-5l3k krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:19:46.886106 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35083
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:47.234194 17164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:47.234566 17164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:47.234745 17164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:47.242246 17164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:47.242455 17164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:47.242578 17164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:47.242682 17164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:47.242775 17164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:47.252585 17164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:47.254145 17164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:47.255720 17164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:47.267823 17169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:47.267742 17170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:47.267943 17173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:47.268751 17164 server_base.cc:1034] running on GCE node
I20260807 11:19:47.269517 17164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:47.270936 17164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:47.272181 17164 hybrid_clock.cc:648] HybridClock initialized: now 1786101587272133 us; error 40 us; skew 500 ppm
I20260807 11:19:47.272532 17164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:47 dist-test-slave-5l3k krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:47.279189 17164 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:47.280761 17164 webserver.cc:460] Webserver started at http://127.2.223.190:43079/ using document root <none> and password file <none>
I20260807 11:19:47.281450 17164 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:47.281596 17164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:47.281903 17164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:47.286334 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "217a373a56fe479ba7d9d4c25dad1f9c"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "86f036649d4bcbfa008d371338e59c0d"
server_key_iv: "5417b148597707df79ca9d80c3405a6c"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.287569 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "217a373a56fe479ba7d9d4c25dad1f9c"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "86f036649d4bcbfa008d371338e59c0d"
server_key_iv: "5417b148597707df79ca9d80c3405a6c"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.294366 17164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:19:47.297895 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:47.299365 17164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:19:47.299517 17164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "217a373a56fe479ba7d9d4c25dad1f9c"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "86f036649d4bcbfa008d371338e59c0d"
server_key_iv: "5417b148597707df79ca9d80c3405a6c"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.299686 17164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:47.335014 17164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:47.339596 17164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:47.339846 17164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:47.361405 17164 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:35083
I20260807 11:19:47.361436 17237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:35083 every 8 connection(s)
I20260807 11:19:47.363032 17164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:47.363736 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17164
I20260807 11:19:47.363999 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:19:47.364928 2942 external_mini_cluster.cc:1428] Setting key acda1c4eb761e1d02aa71d3912cfb627
I20260807 11:19:47.367704 17238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:47.374150 17238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:19:47 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.223.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.189@KRBTEST.COM" created.
I20260807 11:19:47.399344 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.377286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35083 (local address 127.0.0.1:39514)
0807 11:19:47.378324 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.378380 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:47.381234 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:47.382096 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:47.382106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:47.383099 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:47.383595 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.383621 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.385174 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.385182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.387134 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.387153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.388093 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.388103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.388332 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.389033 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:47.389066 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:47.391773 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:47.395101 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.395114 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.395125 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.395545 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.395938 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.395944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.395947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.396105 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.396894 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:47.396902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:47.397109 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 11:19:47.397364 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 11:19:47.397987 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":139,"thread_start_us":438,"threads_started":1}
I20260807 11:19:47.399323 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.377286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.2.223.190:35083)
0807 11:19:47.379264 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:47.379270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:47.379311 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:19:47.381635 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:47.381641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:47.381855 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:47.382016 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:47.383807 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.385005 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.387280 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.387830 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.387961 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.392014 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:47.392023 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:47.392033 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:47.392065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:47.394980 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.395677 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.395684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.395686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.395830 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.396223 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.396233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.396235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.396751 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:47.396958 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:47.397498 (+ 540us) server_negotiation.cc:299] Negotiation successful
0807 11:19:47.397835 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1785,"spinlock_wait_cycles":38784,"thread_start_us":1554,"threads_started":1}
I20260807 11:19:47.402396 17238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:19:47.403671 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.403173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52880)
0807 11:19:47.403366 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.403506 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:19:47.403965 17182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
I20260807 11:19:47.405803 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.405328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52882)
0807 11:19:47.405544 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.405636 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 11:19:47.406013 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:42581: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
I20260807 11:19:47.431802 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:42581
--webserver_interface=127.2.223.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:19:47.455439 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } attempt: 1
I20260807 11:19:47.456712 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.456227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52890)
0807 11:19:47.456443 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.456574 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 11:19:47.457749 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.457402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52902)
0807 11:19:47.457541 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.457613 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 11:19:47.457922 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:42581: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
I20260807 11:19:47.526214 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } attempt: 2
I20260807 11:19:47.527433 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.526935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52912)
0807 11:19:47.527204 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.527295 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:19:47.528486 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.528137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52914)
0807 11:19:47.528282 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.528357 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:19:47.528677 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:42581: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
I20260807 11:19:47.619927 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } attempt: 3
I20260807 11:19:47.621100 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.620645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52918)
0807 11:19:47.620849 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.620960 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 11:19:47.622115 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.621740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52920)
0807 11:19:47.621893 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.621960 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 11:19:47.622339 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:42581: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
W20260807 11:19:47.765426 17245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260807 11:19:47.765561 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } attempt: 4
W20260807 11:19:47.765758 17245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:47.765918 17245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 11:19:47.766814 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.766216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52924)
0807 11:19:47.766598 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.766684 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260807 11:19:47.767781 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.767434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52936)
0807 11:19:47.767574 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.767667 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 11:19:47.767952 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:42581: Network error: Client connection negotiation failed: client connection to 127.2.223.189:42581: connect: Connection refused (error 111)
W20260807 11:19:47.773217 17245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:47.773345 17245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:47.773464 17245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:47.773566 17245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:47.773640 17245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:47.783766 17245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.223.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:47.785135 17245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:47.786451 17245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:47.798760 17251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:47.798712 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:47.798729 17252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:47.799544 17245 server_base.cc:1034] running on GCE node
I20260807 11:19:47.800307 17245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:47.802016 17245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:47.803232 17245 hybrid_clock.cc:648] HybridClock initialized: now 1786101587803171 us; error 52 us; skew 500 ppm
I20260807 11:19:47.803584 17245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:47 dist-test-slave-5l3k krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:47.806766 17245 init.cc:377] Logged in from keytab as kudu/127.2.223.189@KRBTEST.COM (short username kudu)
I20260807 11:19:47.808283 17245 webserver.cc:460] Webserver started at http://127.2.223.189:38511/ using document root <none> and password file <none>
I20260807 11:19:47.809020 17245 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:47.809122 17245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:47.809458 17245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:47.812510 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data/instance:
uuid: "9263fde1b60d470f8c33780b7278d3fb"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "506ae41ea87a1a38895c1a5b7dbf9f27"
server_key_iv: "b576ffc906d3aebea702945d385c0826"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.813320 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal/instance:
uuid: "9263fde1b60d470f8c33780b7278d3fb"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "506ae41ea87a1a38895c1a5b7dbf9f27"
server_key_iv: "b576ffc906d3aebea702945d385c0826"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.818457 17245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:19:47.822290 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:47.823644 17245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:19:47.823822 17245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "9263fde1b60d470f8c33780b7278d3fb"
format_stamp: "Formatted at 2026-08-07 11:19:47 on dist-test-slave-5l3k"
server_key: "506ae41ea87a1a38895c1a5b7dbf9f27"
server_key_iv: "b576ffc906d3aebea702945d385c0826"
server_key_version: "encryptionkey@0"
I20260807 11:19:47.824016 17245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:47.854732 17245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:47.859284 17245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:47.859606 17245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:47.879501 17245 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.189:42581
I20260807 11:19:47.879541 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.189:42581 every 8 connection(s)
I20260807 11:19:47.880851 17245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
I20260807 11:19:47.884569 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:47.888532 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17245
I20260807 11:19:47.888757 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-1/wal/instance
I20260807 11:19:47.889343 2942 external_mini_cluster.cc:1428] Setting key 7a40ce3482503012a37630715795b50d
I20260807 11:19:47.892933 17319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:19:47 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:47.915902 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.896619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.2.223.190:35083)
0807 11:19:47.896887 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:47.896895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:47.898082 (+ 1187us) server_negotiation.cc:407] Connection header received
0807 11:19:47.899996 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:47.900004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:47.900103 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:47.900226 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:47.901659 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.902770 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.905311 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.905749 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.905845 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.910070 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:47.910075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:47.910078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:47.910114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:47.912797 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.913560 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.913570 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.913573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.913696 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.914119 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.914125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.914128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.914417 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:47.914623 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:47.915328 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 11:19:47.915570 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:19:47.917133 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.897058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35083 (local address 127.0.0.1:50412)
0807 11:19:47.897898 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.897946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:47.899730 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:47.900422 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:47.900430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:47.900932 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:47.901486 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.901502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.902951 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.902958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.905156 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.905191 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.905971 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.905980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.906305 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.907102 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:47.907135 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:47.909829 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:47.912952 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.912961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.912977 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.913428 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.913809 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.913814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.913818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.913999 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.914581 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:47.914590 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:47.914932 (+ 342us) client_negotiation.cc:769] Sending connection context
0807 11:19:47.915232 (+ 300us) client_negotiation.cc:240] Negotiation successful
0807 11:19:47.915743 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":371,"threads_started":1}
I20260807 11:19:47.918948 17319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.223.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.188@KRBTEST.COM" created.
Aug 07 11:19:47 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:19:47.933758 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.919833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52938)
0807 11:19:47.920047 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.920066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:47.920168 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:47.921119 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:47.921123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:47.921463 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:47.921604 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.921612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.923116 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.923121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.923989 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:47.924000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.924778 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.924784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:47.925016 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.925585 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:47.925600 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:47.928068 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:47.931215 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.931222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.931229 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.931756 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.932122 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:47.932129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:47.932135 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:47.932310 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:47.932900 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:47.932907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:47.933079 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 11:19:47.933236 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 11:19:47.933503 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":36}
I20260807 11:19:47.934214 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.919948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.2.223.189:42581)
0807 11:19:47.920733 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:47.920739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:47.920764 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:19:47.920806 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:47.920811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:47.920975 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:47.921088 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:47.921736 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.922980 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.924124 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:47.924638 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:47.924770 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:47.928303 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:47.928312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:47.928324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:47.928351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:47.931086 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.931874 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.931880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.931882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.932003 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:47.932431 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:47.932434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:47.932435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:47.932779 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:47.932917 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:47.933399 (+ 482us) server_negotiation.cc:299] Negotiation successful
0807 11:19:47.933685 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":632,"thread_start_us":516,"threads_started":1}
I20260807 11:19:47.936826 17319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:19:47.937978 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.937538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55068)
0807 11:19:47.937695 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.937841 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:19:47.938225 17264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:47.940011 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.939545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55082)
0807 11:19:47.939753 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.939863 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:19:47.940289 17319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
I20260807 11:19:47.957458 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:46017
--webserver_interface=127.2.223.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:19:47.989611 17319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 1
I20260807 11:19:47.990983 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.990437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55088)
0807 11:19:47.990698 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.990808 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260807 11:19:47.991950 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:47.991626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55094)
0807 11:19:47.991781 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:47.991845 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 11:19:47.992833 17319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.061105 17319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 2
I20260807 11:19:48.062325 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.061835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55100)
0807 11:19:48.062074 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.062164 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 11:19:48.063442 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.063119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55104)
0807 11:19:48.063262 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.063333 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 11:19:48.063710 17319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.067178 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } attempt: 5
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:19:48.080044 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.067831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.2.223.189:42581)
0807 11:19:48.068020 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.068026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.068051 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:19:48.068202 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.068207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.068287 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.068412 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.069156 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.069901 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.071152 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.071599 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.071730 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.074759 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.074764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.074767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.074799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.077431 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.078102 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.078108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.078112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.078245 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.078670 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.078675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.078678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.078959 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.079115 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.079525 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.079739 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:19:48.080057 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.067755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52952)
0807 11:19:48.067921 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.067944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.068053 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.068433 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.068436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.068809 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.069006 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.069014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.070083 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.070087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.071007 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.071027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.071744 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.071750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.071924 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.072534 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.072552 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.074568 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.077586 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.077592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.077597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.077984 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.078357 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.078362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.078366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.078559 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.079082 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.079087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.079256 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.079402 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.079677 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 11:19:48.081164 17238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:19:48.082156 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.081779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55110)
0807 11:19:48.081937 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.082026 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 11:19:48.083015 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.082717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55112)
0807 11:19:48.082859 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.082928 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 11:19:48.083235 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.134464 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 1
I20260807 11:19:48.135789 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.135313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55114)
0807 11:19:48.135526 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.135625 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:19:48.136799 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.136442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55120)
0807 11:19:48.136612 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.136685 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 11:19:48.137020 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.154924 17319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 3
I20260807 11:19:48.156169 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.155708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55136)
0807 11:19:48.155908 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.155997 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 11:19:48.157296 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.156859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55138)
0807 11:19:48.157109 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.157181 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260807 11:19:48.157487 17319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.205235 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 2
I20260807 11:19:48.206414 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.205872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55142)
0807 11:19:48.206117 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.206234 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 11:19:48.207414 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.207054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55148)
0807 11:19:48.207227 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.207300 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 11:19:48.207645 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
W20260807 11:19:48.296418 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:48.296762 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:48.296952 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 11:19:48.300683 17319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 4
I20260807 11:19:48.301785 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.301297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55162)
0807 11:19:48.301534 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.301622 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 11:19:48.302801 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.302395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55166)
0807 11:19:48.302564 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.302653 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 11:19:48.303010 17319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
W20260807 11:19:48.304833 17326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:48.304947 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:48.305042 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:48.305143 17326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:48.305219 17326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:48.313836 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 3
I20260807 11:19:48.314894 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.314437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55170)
0807 11:19:48.314666 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.314750 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:19:48.315413 17326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35083,127.2.223.189:42581,127.2.223.188:46017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.223.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:48.316048 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.315545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55186)
0807 11:19:48.315819 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.315911 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260807 11:19:48.316288 17238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:46017: Network error: Client connection negotiation failed: client connection to 127.2.223.188:46017: connect: Connection refused (error 111)
I20260807 11:19:48.316985 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:48.318307 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:48.330986 17332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:48.331314 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:48.331400 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:48.332334 17326 server_base.cc:1034] running on GCE node
I20260807 11:19:48.333094 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:48.334470 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:48.335740 17326 hybrid_clock.cc:648] HybridClock initialized: now 1786101588335693 us; error 42 us; skew 500 ppm
I20260807 11:19:48.336079 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:48.339340 17326 init.cc:377] Logged in from keytab as kudu/127.2.223.188@KRBTEST.COM (short username kudu)
I20260807 11:19:48.340736 17326 webserver.cc:460] Webserver started at http://127.2.223.188:46419/ using document root <none> and password file <none>
I20260807 11:19:48.341409 17326 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:48.341552 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:48.341864 17326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:48.345000 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data/instance:
uuid: "91067c26331a4d50b166355ecf33d7be"
format_stamp: "Formatted at 2026-08-07 11:19:48 on dist-test-slave-5l3k"
server_key: "cd93ebcd2143f81002e21f8e99ea35a4"
server_key_iv: "24c342f1fcedc35b326aacc4fe5ee53b"
server_key_version: "encryptionkey@0"
I20260807 11:19:48.345923 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal/instance:
uuid: "91067c26331a4d50b166355ecf33d7be"
format_stamp: "Formatted at 2026-08-07 11:19:48 on dist-test-slave-5l3k"
server_key: "cd93ebcd2143f81002e21f8e99ea35a4"
server_key_iv: "24c342f1fcedc35b326aacc4fe5ee53b"
server_key_version: "encryptionkey@0"
I20260807 11:19:48.350917 17326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:19:48.354115 17343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:48.355409 17326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:19:48.355624 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "91067c26331a4d50b166355ecf33d7be"
format_stamp: "Formatted at 2026-08-07 11:19:48 on dist-test-slave-5l3k"
server_key: "cd93ebcd2143f81002e21f8e99ea35a4"
server_key_iv: "24c342f1fcedc35b326aacc4fe5ee53b"
server_key_version: "encryptionkey@0"
I20260807 11:19:48.355858 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:48.374451 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:48.378469 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:48.378805 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:48.397902 17326 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.188:46017
I20260807 11:19:48.397933 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.188:46017 every 8 connection(s)
I20260807 11:19:48.399294 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
I20260807 11:19:48.403297 17399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:48.405313 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17326
I20260807 11:19:48.405545 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786101369622807-2942-0/minicluster-data/master-2/wal/instance
I20260807 11:19:48.406234 2942 external_mini_cluster.cc:1428] Setting key e7b9c1e70b69d23a28c835a4b3c01f8e
I20260807 11:19:48.413786 17399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:48.430884 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.408444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.2.223.188:46017)
0807 11:19:48.409498 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.409513 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.409568 (+ 55us) server_negotiation.cc:407] Connection header received
0807 11:19:48.411230 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.411260 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.411655 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.411962 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.413845 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.415278 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.417145 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.417982 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.418170 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.422890 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.422899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.422911 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.422954 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.426330 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.427072 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.427084 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.427089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.427239 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.427664 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.427671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.427676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.428401 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.428700 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.429201 (+ 501us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.429602 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":564,"threads_started":1}
I20260807 11:19:48.433010 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.416503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.2.223.190:35083)
0807 11:19:48.417075 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.417093 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.417604 (+ 511us) server_negotiation.cc:407] Connection header received
0807 11:19:48.417801 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.417807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.417892 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.418019 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.419481 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.420202 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.421883 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.422623 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.422761 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.426626 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.426630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.426633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.426673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.430242 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.430959 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.430966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.430971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.431137 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.431560 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.431566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.431568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.431913 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.432076 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.432579 (+ 503us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.432744 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
I20260807 11:19:48.433205 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.416551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35083 (local address 127.0.0.1:50418)
0807 11:19:48.417456 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.417501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.417662 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.418088 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.418095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.418914 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.419318 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.419333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.420365 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.420373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.421739 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.421758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.422890 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.422898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.423110 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.423753 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.423777 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.426406 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.430364 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.430377 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.430382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.430836 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.431269 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.431274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.431278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.431438 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.432034 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.432039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.432275 (+ 236us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.432472 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.432765 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":529,"threads_started":1}
I20260807 11:19:48.434576 17399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:48.447885 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.435422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:42581 (local address 127.0.0.1:52960)
0807 11:19:48.435572 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.435588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.435704 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.436007 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.436011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.436457 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.436632 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.436643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.437761 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.437765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.438751 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.438766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.439537 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.439541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.439675 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.440215 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.440248 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.442290 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.445449 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.445453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.445456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.445820 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.446233 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.446236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.446238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.446365 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.446976 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.446994 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.447178 (+ 184us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.447338 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.447613 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 11:19:48.447963 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.435334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.2.223.189:42581)
0807 11:19:48.435501 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.435507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.435659 (+ 152us) server_negotiation.cc:407] Connection header received
0807 11:19:48.435814 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.435819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.435889 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.435987 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.436790 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.437606 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.438883 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.439403 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.439536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.442521 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.442529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.442534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.442573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.445337 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.445937 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.445945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.445950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.446120 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.446472 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.446500 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.446503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.446865 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.447035 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.447412 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.447564 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 11:19:48.449155 17399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:19:48.449676 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.434157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.2.223.190:35083)
0807 11:19:48.434300 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.434304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.434519 (+ 215us) server_negotiation.cc:407] Connection header received
0807 11:19:48.434678 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.434683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.434767 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.434863 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.436307 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.437148 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.438523 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.439087 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.439175 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.443876 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.443880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.443883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.443912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.446927 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.447711 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.447717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.447722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.447862 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.448346 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.448352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.448355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.448698 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.448987 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.449163 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.449379 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:19:48.462235 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.449875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55204)
0807 11:19:48.450226 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.450244 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.450362 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.450698 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.450702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.451115 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.451310 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.451323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.452240 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.452248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.453077 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.453087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.453792 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.453797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.453979 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.454538 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.454558 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.456614 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.459881 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.459888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.459892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.460346 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.460713 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.460717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.460719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.460852 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.461503 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.461506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.461621 (+ 115us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.461728 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.461931 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260807 11:19:48.462705 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.449875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.2.223.188:46017)
0807 11:19:48.450099 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.450103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.450327 (+ 224us) server_negotiation.cc:407] Connection header received
0807 11:19:48.450462 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.450465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.450562 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.450645 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.451453 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.452123 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.453213 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.453656 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.453775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.456790 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.456796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.456800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.456833 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.459778 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.460459 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.460467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.460470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.460612 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.460956 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.460961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.460964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.461352 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.461527 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.461852 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.462110 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260807 11:19:48.466543 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.451418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.2.223.189:42581)
0807 11:19:48.451591 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.451597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.451733 (+ 136us) server_negotiation.cc:407] Connection header received
0807 11:19:48.451898 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.451902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.451979 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.452071 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.453325 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.454000 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.455317 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.455959 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.456098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.460683 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.460689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.460693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.460730 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.463967 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.464603 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.464609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.464612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.464753 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.465139 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.465146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.465149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.465405 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.465699 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.465957 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.466168 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 11:19:48.467865 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: Bootstrap starting.
I20260807 11:19:48.471362 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:19:48.471876 17399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:48.472888 17399 log.cc:824] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:48.475865 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: No bootstrap required, opened a new log
I20260807 11:19:48.487562 17399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.488207 17399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:48.488365 17399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91067c26331a4d50b166355ecf33d7be, State: Initialized, Role: FOLLOWER
I20260807 11:19:48.489310 17399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.490711 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.491277 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.492159 17399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:48.493710 17238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 4
I20260807 11:19:48.499539 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.477123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.2.223.189:42581)
0807 11:19:48.477532 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.477538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.478654 (+ 1116us) server_negotiation.cc:407] Connection header received
0807 11:19:48.478729 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.478736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.478831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.478933 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.481926 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.483384 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.486105 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.487070 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.487198 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.491861 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.491866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.491869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.491904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.495370 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.496517 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.496526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.496532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.496735 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.497252 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.497257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.497261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.497642 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.497900 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.498941 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.499183 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260807 11:19:48.499740 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.477077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.2.223.188:46017)
0807 11:19:48.477285 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.477291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.477433 (+ 142us) server_negotiation.cc:407] Connection header received
0807 11:19:48.477707 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.477713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.477816 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.477926 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.480415 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.481594 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.483768 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.484486 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.484582 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.488898 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.488904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.488908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.488948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.492790 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.497130 (+ 4340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.497135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.497140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.497333 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.497823 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.497828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.497831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.498264 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.498525 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.499064 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.499308 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:19:48.512841 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.476566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.2.223.190:35083)
0807 11:19:48.476775 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.476781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.493032 (+ 16251us) server_negotiation.cc:407] Connection header received
0807 11:19:48.493327 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.493335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.493440 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.493535 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.495897 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.497188 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.499585 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.500242 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.500333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.505911 (+ 5578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.505919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.505926 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.505967 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.509446 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.510359 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.510367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.510371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.510568 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.511096 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.511101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.511105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.511491 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.511769 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.512210 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.512440 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:19:48.516482 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.494293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55224)
0807 11:19:48.494657 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.494677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.494775 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.498072 (+ 3297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.498076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.498616 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.498831 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.498847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.502539 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.502546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.503935 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.503948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.504711 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.504717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.504903 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.505433 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.505451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.508652 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.513119 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.513128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.513143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.513840 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.514277 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.514285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.514291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.514542 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.515343 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.515351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.515558 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.515738 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.516086 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260807 11:19:48.516563 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.494977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.2.223.188:46017)
0807 11:19:48.497064 (+ 2087us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.497070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.497092 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:19:48.497140 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.497145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.497216 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.499011 (+ 1795us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.499346 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.500602 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.504086 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.504564 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.504698 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.508888 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.508897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.508905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.508952 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.512965 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.513990 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.513995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.513997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.514151 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.514703 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.514709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.514711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.515108 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.515346 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.515900 (+ 554us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.516118 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":1774,"threads_started":1}
I20260807 11:19:48.524325 17238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: Bootstrap starting.
I20260807 11:19:48.517683 17326 master_runner.cc:186] Error getting master registration for 127.2.223.190:35083: OK, instance_id { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" instance_seqno: 1786101587355305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:19:48.526872 17326 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 11:19:48.528026 17425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:19:48.528163 17425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:19:48.529193 17238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:48.530755 17238 log.cc:824] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:48.534026 17238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: No bootstrap required, opened a new log
I20260807 11:19:48.546212 17238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.546665 17238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:48.546801 17238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217a373a56fe479ba7d9d4c25dad1f9c, State: Initialized, Role: FOLLOWER
I20260807 11:19:48.547487 17238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.548436 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.548792 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.549961 17238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:19:48.560431 17440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:19:48.560621 17440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:19:48.559466 17164 master_runner.cc:186] Error getting master registration for 127.2.223.189:42581: OK, instance_id { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" instance_seqno: 1786101587873955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:19:48.572084 17164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:19:48.582924 17428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:48.583245 17428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.585029 17428 leader_election.cc:288] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9263fde1b60d470f8c33780b7278d3fb (127.2.223.189:42581), 91067c26331a4d50b166355ecf33d7be (127.2.223.188:46017)
I20260807 11:19:48.586081 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91067c26331a4d50b166355ecf33d7be" is_pre_election: true
I20260807 11:19:48.586856 17374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a373a56fe479ba7d9d4c25dad1f9c in term 0.
I20260807 11:19:48.588207 17183 leader_election.cc:302] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a373a56fe479ba7d9d4c25dad1f9c, 91067c26331a4d50b166355ecf33d7be; no voters:
I20260807 11:19:48.588724 17428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:19:48.588887 17428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:19:48.589011 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:48.592525 17428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
W20260807 11:19:48.592780 17186 tablet.cc:2369] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:48.593745 17428 leader_election.cc:288] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [CANDIDATE]: Term 1 election: Requested vote from peers 9263fde1b60d470f8c33780b7278d3fb (127.2.223.189:42581), 91067c26331a4d50b166355ecf33d7be (127.2.223.188:46017)
I20260807 11:19:48.594718 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91067c26331a4d50b166355ecf33d7be"
I20260807 11:19:48.594947 17374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:48.598428 17374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a373a56fe479ba7d9d4c25dad1f9c in term 1.
I20260807 11:19:48.598943 17183 leader_election.cc:302] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a373a56fe479ba7d9d4c25dad1f9c, 91067c26331a4d50b166355ecf33d7be; no voters:
I20260807 11:19:48.599344 17428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:48.602331 17319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } attempt: 5
I20260807 11:19:48.604097 17428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [term 1 LEADER]: Becoming Leader. State: Replica: 217a373a56fe479ba7d9d4c25dad1f9c, State: Running, Role: LEADER
I20260807 11:19:48.605108 17428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
Aug 07 11:19:48 dist-test-slave-5l3k krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101587, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:19:48.616644 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217a373a56fe479ba7d9d4c25dad1f9c. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.617415 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:48.623379 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.603091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:46017 (local address 127.0.0.1:55234)
0807 11:19:48.603426 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:48.603446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:48.603574 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:48.604787 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:48.604792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:48.605434 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:48.605667 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.605682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.607708 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.607715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.609351 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:48.609369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.610916 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.610923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:48.611187 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.612203 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:48.612231 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:48.615385 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:48.620123 (+ 4738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.620129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.620134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.620729 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.621246 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:48.621251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:48.621256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:48.621429 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:48.622210 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:48.622218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:48.622395 (+ 177us) client_negotiation.cc:769] Sending connection context
0807 11:19:48.622583 (+ 188us) client_negotiation.cc:240] Negotiation successful
0807 11:19:48.622901 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260807 11:19:48.624496 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:48.603882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.2.223.188:46017)
0807 11:19:48.604125 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:48.604132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:48.604153 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:19:48.604210 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:48.604215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:48.604316 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:48.604435 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:48.605997 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.607367 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.609733 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:48.610525 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:48.610662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:48.615631 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:48.615639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:48.615644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:48.615691 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:48.619934 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.620886 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.620892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.620896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.621088 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:48.621570 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:48.621576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:48.621580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:48.621967 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:48.622128 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:48.623820 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0807 11:19:48.624087 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 11:19:48.630791 17349 tablet.cc:2369] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:48.632433 17319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: Bootstrap starting.
I20260807 11:19:48.636776 17319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:48.638226 17319 log.cc:824] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:48.642284 17319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: No bootstrap required, opened a new log
I20260807 11:19:48.642807 17444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:48.644811 17444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:48.655949 17319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.656405 17319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:48.656558 17319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9263fde1b60d470f8c33780b7278d3fb, State: Initialized, Role: FOLLOWER
I20260807 11:19:48.660787 17319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } }
I20260807 11:19:48.663410 17374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [term 1 FOLLOWER]: Refusing update from remote peer 217a373a56fe479ba7d9d4c25dad1f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:48.668012 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.668499 17428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:48.670434 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.670308 17319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:48.671278 17287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:48.681721 17288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9263fde1b60d470f8c33780b7278d3fb"
I20260807 11:19:48.682737 17289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9263fde1b60d470f8c33780b7278d3fb" is_pre_election: true
I20260807 11:19:48.686403 17287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [term 1 FOLLOWER]: Refusing update from remote peer 217a373a56fe479ba7d9d4c25dad1f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:48.692570 17428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:48.702804 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217a373a56fe479ba7d9d4c25dad1f9c. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.703300 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.716759 17446 mvcc.cc:205] Tried to move back new op lower bound from 7315872107149217792 to 7315872106935443456. Current Snapshot: MvccSnapshot[applied={T|T < 7315872107149217792}]
I20260807 11:19:48.724138 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.724566 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.727947 17444 catalog_manager.cc:1347] Generated new cluster ID: cb6677e58fb1475585b71442d37eef87
I20260807 11:19:48.728044 17444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:48.734123 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217a373a56fe479ba7d9d4c25dad1f9c. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.734472 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.738036 17454 mvcc.cc:205] Tried to move back new op lower bound from 7315872107149217792 to 7315872106935443456. Current Snapshot: MvccSnapshot[applied={T|T < 7315872107149217792}]
I20260807 11:19:48.739998 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.740489 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:48.741031 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.741379 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:48.754765 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217a373a56fe479ba7d9d4c25dad1f9c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35083 } } peers { permanent_uuid: "9263fde1b60d470f8c33780b7278d3fb" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 42581 } } peers { permanent_uuid: "91067c26331a4d50b166355ecf33d7be" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 46017 } } }
I20260807 11:19:48.755138 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:19:48.793471 17444 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:19:48.795463 17444 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:48.829700 17444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217a373a56fe479ba7d9d4c25dad1f9c: Generated new TSK 0
I20260807 11:19:48.830834 17444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:19:48.836313 17466 catalog_manager.cc:1259] Loaded cluster ID: cb6677e58fb1475585b71442d37eef87
I20260807 11:19:48.836503 17466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: loading cluster ID for follower catalog manager: success
I20260807 11:19:48.840307 17466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: acquiring CA information for follower catalog manager: success
I20260807 11:19:48.842903 17466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9263fde1b60d470f8c33780b7278d3fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:19:49.530256 17425 catalog_manager.cc:1259] Loaded cluster ID: cb6677e58fb1475585b71442d37eef87
I20260807 11:19:49.530457 17425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: loading cluster ID for follower catalog manager: success
I20260807 11:19:49.532975 17425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: acquiring CA information for follower catalog manager: success
I20260807 11:19:49.534767 17425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91067c26331a4d50b166355ecf33d7be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:19:50.001179 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17164
I20260807 11:19:50.002604 17232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:50.116197 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17164
I20260807 11:19:50.142194 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17245
I20260807 11:19:50.143356 17314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:50.251606 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17245
I20260807 11:19:50.276762 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17326
I20260807 11:19:50.277956 17394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:50.388684 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17326
2026-08-07T11:19:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4948 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:19:50 dist-test-slave-5l3k krb5kdc[17479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:19:50 dist-test-slave-5l3k krb5kdc[17479](info): set up 2 sockets
Aug 07 11:19:50 dist-test-slave-5l3k krb5kdc[17479](info): commencing operation
krb5kdc: starting...
W20260807 11:19:51.563807 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:19:51 dist-test-slave-5l3k krb5kdc[17479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:19:51.858202 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33629
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:52.202720 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:52.203081 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:52.203240 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:52.210525 17494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:52.210670 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:52.210777 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:52.210877 17494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:52.210952 17494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:52.221035 17494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:33629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:52.222450 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:52.223917 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:52.236058 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:52.236058 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:52.236055 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:52.237327 17494 server_base.cc:1034] running on GCE node
I20260807 11:19:52.238127 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:52.239694 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:52.240926 17494 hybrid_clock.cc:648] HybridClock initialized: now 1786101592240842 us; error 77 us; skew 500 ppm
I20260807 11:19:52.241277 17494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:52 dist-test-slave-5l3k krb5kdc[17479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:52.244618 17494 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:52.246192 17494 webserver.cc:460] Webserver started at http://127.2.223.190:38347/ using document root <none> and password file <none>
I20260807 11:19:52.246961 17494 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:52.247116 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:52.247463 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:52.250742 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "75a8e66f6061451da44acb730f84e9f6"
format_stamp: "Formatted at 2026-08-07 11:19:52 on dist-test-slave-5l3k"
server_key: "c04da63b10d214c1f108e19442e1a09a"
server_key_iv: "86d32e62f9eaa5fbeeac313236c67d2e"
server_key_version: "encryptionkey@0"
I20260807 11:19:52.251538 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "75a8e66f6061451da44acb730f84e9f6"
format_stamp: "Formatted at 2026-08-07 11:19:52 on dist-test-slave-5l3k"
server_key: "c04da63b10d214c1f108e19442e1a09a"
server_key_iv: "86d32e62f9eaa5fbeeac313236c67d2e"
server_key_version: "encryptionkey@0"
I20260807 11:19:52.256978 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:19:52.260583 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:52.261962 17494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:19:52.262140 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "75a8e66f6061451da44acb730f84e9f6"
format_stamp: "Formatted at 2026-08-07 11:19:52 on dist-test-slave-5l3k"
server_key: "c04da63b10d214c1f108e19442e1a09a"
server_key_iv: "86d32e62f9eaa5fbeeac313236c67d2e"
server_key_version: "encryptionkey@0"
I20260807 11:19:52.262349 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:52.273469 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:52.278038 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:52.278353 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:52.297828 17494 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:33629
I20260807 11:19:52.297837 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:33629 every 8 connection(s)
I20260807 11:19:52.299861 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:52.304267 17563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:52.308765 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17494
I20260807 11:19:52.308995 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:19:52.309695 2942 external_mini_cluster.cc:1428] Setting key ea678c113af83eebdb22cbbe68cb8ab0
I20260807 11:19:52.316519 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6: Bootstrap starting.
I20260807 11:19:52.320355 17563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:52.321483 17563 log.cc:824] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:52.324484 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6: No bootstrap required, opened a new log
Aug 07 11:19:52 dist-test-slave-5l3k krb5kdc[17479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:52.331588 17563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } }
I20260807 11:19:52.331947 17563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:52.332050 17563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a8e66f6061451da44acb730f84e9f6, State: Initialized, Role: FOLLOWER
I20260807 11:19:52.332587 17563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } }
I20260807 11:19:52.332813 17563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:19:52.332969 17563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:19:52.333153 17563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:52.335654 17563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } }
I20260807 11:19:52.336134 17563 leader_election.cc:302] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a8e66f6061451da44acb730f84e9f6; no voters:
I20260807 11:19:52.336782 17563 leader_election.cc:288] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:19:52.336886 17568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:52.338786 17568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [term 1 LEADER]: Becoming Leader. State: Replica: 75a8e66f6061451da44acb730f84e9f6, State: Running, Role: LEADER
I20260807 11:19:52.339263 17563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:52.339352 17568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } }
I20260807 11:19:52.341892 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:52.311636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.2.223.190:33629)
0807 11:19:52.312531 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:52.312548 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:52.312601 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:19:52.315109 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:52.315139 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:52.315627 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:52.316078 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:52.317506 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:52.319652 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:52.321060 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:52.322181 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:52.322350 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:52.333064 (+ 10714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:52.333078 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:52.333112 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:52.333157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:52.336423 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:52.337309 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:52.337319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:52.337329 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:52.337568 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:52.338080 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:52.338085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:52.338088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:52.338772 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:52.339096 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:52.339883 (+ 787us) server_negotiation.cc:299] Negotiation successful
0807 11:19:52.340331 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":376,"threads_started":1}
I20260807 11:19:52.347854 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:19:52.349931 17569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a8e66f6061451da44acb730f84e9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } } }
I20260807 11:19:52.350383 17569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:52.351722 17570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a8e66f6061451da44acb730f84e9f6. Latest consensus state: current_term: 1 leader_uuid: "75a8e66f6061451da44acb730f84e9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a8e66f6061451da44acb730f84e9f6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 33629 } } }
I20260807 11:19:52.352137 17570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:52.353080 17577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:52.358701 17577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:52.367944 17577 catalog_manager.cc:1347] Generated new cluster ID: 681a8e4ecf37402aa6b53554db15be79
I20260807 11:19:52.368070 17577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:52.382547 17577 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:19:52.383796 17577 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:52.392838 17577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a8e66f6061451da44acb730f84e9f6: Generated new TSK 0
I20260807 11:19:52.393724 17577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:19:52.410874 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17494
I20260807 11:19:52.411984 17558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:52.503835 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17494
2026-08-07T11:19:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2107 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:19:52 dist-test-slave-5l3k krb5kdc[17593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:19:52 dist-test-slave-5l3k krb5kdc[17593](info): set up 2 sockets
Aug 07 11:19:52 dist-test-slave-5l3k krb5kdc[17593](info): commencing operation
krb5kdc: starting...
W20260807 11:19:53.667929 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.009s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:19:53 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:19:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:19:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:19:53.982302 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:54.324600 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:54.324900 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:54.325098 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:54.332279 17607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:54.332381 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:54.332499 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:54.332603 17607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:54.332671 17607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:54.342612 17607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:54.344022 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:54.345423 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:54.358714 17614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:54.358714 17617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:54.358675 17615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:54.359364 17607 server_base.cc:1034] running on GCE node
I20260807 11:19:54.360265 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:54.362808 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:54.364042 17607 hybrid_clock.cc:648] HybridClock initialized: now 1786101594363995 us; error 34 us; skew 500 ppm
I20260807 11:19:54.364392 17607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:54 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:54.367610 17607 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:54.369084 17607 webserver.cc:460] Webserver started at http://127.2.223.190:43495/ using document root <none> and password file <none>
I20260807 11:19:54.369767 17607 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:54.369918 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:54.370240 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:54.373396 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "e7d58cb3825c4dd388e08a583385853b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "6387abecb9530524fa223e38ecfb9b73"
server_key_iv: "79a56fa2d54ddc8bee66bf67c9e1f8d1"
server_key_version: "encryptionkey@0"
I20260807 11:19:54.374167 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "e7d58cb3825c4dd388e08a583385853b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "6387abecb9530524fa223e38ecfb9b73"
server_key_iv: "79a56fa2d54ddc8bee66bf67c9e1f8d1"
server_key_version: "encryptionkey@0"
I20260807 11:19:54.379438 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:19:54.382963 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:54.384253 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:19:54.384431 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "e7d58cb3825c4dd388e08a583385853b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "6387abecb9530524fa223e38ecfb9b73"
server_key_iv: "79a56fa2d54ddc8bee66bf67c9e1f8d1"
server_key_version: "encryptionkey@0"
I20260807 11:19:54.384629 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:54.398378 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:54.402706 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:54.402980 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:54.422744 17607 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:39045
I20260807 11:19:54.422827 17684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:39045 every 8 connection(s)
I20260807 11:19:54.424170 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:54.427934 17685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:54.434121 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17607
I20260807 11:19:54.434360 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:19:54.435011 2942 external_mini_cluster.cc:1428] Setting key 49ad81c693792f0ed0081412c6d1b159
I20260807 11:19:54.438566 17685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap starting.
I20260807 11:19:54.441967 17685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:54.443128 17685 log.cc:824] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:54.445796 17685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: No bootstrap required, opened a new log
Aug 07 11:19:54 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:54.453843 17685 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:54.454231 17685 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:54.454340 17685 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Initialized, Role: FOLLOWER
I20260807 11:19:54.455119 17685 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:54.455363 17685 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:19:54.455523 17685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:19:54.455706 17685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:54.457844 17688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:54.437044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.2.223.190:39045)
0807 11:19:54.437911 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:54.437922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:54.437962 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:19:54.439686 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:54.439710 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:54.440078 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:54.440392 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:54.442265 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:54.443767 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:54.445081 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:54.445709 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:54.445911 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:54.450258 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:54.450266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:54.450280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:54.450318 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:54.453466 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:54.454139 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:54.454149 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:54.454159 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:54.454393 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:54.454920 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:54.454923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:54.454925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:54.455579 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:54.455945 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:54.456408 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 11:19:54.456855 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":370,"threads_started":1}
I20260807 11:19:54.458314 17685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:54.458881 17685 leader_election.cc:302] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d58cb3825c4dd388e08a583385853b; no voters:
I20260807 11:19:54.459564 17685 leader_election.cc:288] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:19:54.459750 17690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:54.461002 17690 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 LEADER]: Becoming Leader. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Running, Role: LEADER
I20260807 11:19:54.461602 17690 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:54.461784 17685 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:54.464731 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:19:54.464708 17692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d58cb3825c4dd388e08a583385853b. Latest consensus state: current_term: 1 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:19:54.465142 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:54.465200 17692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:54.466876 17696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:54.471897 17696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:54.483016 17696 catalog_manager.cc:1347] Generated new cluster ID: 65df56d7ffc0460f93828cbe91333d26
I20260807 11:19:54.483191 17696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:54.512437 17696 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:19:54.514233 17696 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:54.520653 17696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Generated new TSK 0
I20260807 11:19:54.521481 17696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:19:54.598615 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:54.943578 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:54.943885 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:54.944062 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:54.951165 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:54.951310 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:54.951501 17711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:19:54.961131 17711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:54.962661 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:54.964087 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:54.977917 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:54.977874 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:54.978410 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:54.978871 17711 server_base.cc:1034] running on GCE node
I20260807 11:19:54.979468 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:54.980590 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:54.981796 17711 hybrid_clock.cc:648] HybridClock initialized: now 1786101594981758 us; error 37 us; skew 500 ppm
I20260807 11:19:54.982120 17711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:54 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:54.985574 17711 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:19:54.987105 17711 webserver.cc:460] Webserver started at http://127.2.223.129:38629/ using document root <none> and password file <none>
I20260807 11:19:54.987815 17711 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:54.987984 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:54.988298 17711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:54.991271 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "c5680442316ae3c88a390db730f3a68e"
server_key_iv: "aca7b3ffb55fcda2a43f48b21ab6092f"
server_key_version: "encryptionkey@0"
I20260807 11:19:54.991981 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "c5680442316ae3c88a390db730f3a68e"
server_key_iv: "aca7b3ffb55fcda2a43f48b21ab6092f"
server_key_version: "encryptionkey@0"
I20260807 11:19:54.997030 17711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:19:55.000429 17727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.001724 17711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:19:55.001906 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "c5680442316ae3c88a390db730f3a68e"
server_key_iv: "aca7b3ffb55fcda2a43f48b21ab6092f"
server_key_version: "encryptionkey@0"
I20260807 11:19:55.002107 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:55.031941 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:55.036325 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:55.036666 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:55.037704 17711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:55.039240 17711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:55.039345 17711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.039492 17711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:55.039574 17711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.074596 17711 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:34803
I20260807 11:19:55.074632 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:34803 every 8 connection(s)
I20260807 11:19:55.075898 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:19:55.080044 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17711
I20260807 11:19:55.080276 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:19:55.081007 2942 external_mini_cluster.cc:1428] Setting key ef422e681b40c9e2a013279d1ad98ca4
Aug 07 11:19:55 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:55.097950 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:55.078538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:43271 (local address 127.2.223.190:39045)
0807 11:19:55.078996 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:55.079001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:55.079981 (+ 980us) server_negotiation.cc:407] Connection header received
0807 11:19:55.082299 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:55.082304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:55.082370 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:55.082513 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:55.084179 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.084888 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.086430 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.086903 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.087047 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:55.091279 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:55.091284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:55.091287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:55.091318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:55.094139 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:55.095140 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:55.095147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:55.095158 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:55.095296 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:55.095819 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:55.095826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:55.095831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:55.096232 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:55.096394 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:55.097328 (+ 934us) server_negotiation.cc:299] Negotiation successful
0807 11:19:55.097515 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260807 11:19:55.099040 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:55.079123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:43271)
0807 11:19:55.079815 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:55.079859 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:55.081980 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:55.082776 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:55.082788 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:55.083432 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:55.083996 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:55.084013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.085043 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.085049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:55.086273 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:55.086286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.087153 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.087161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:55.087412 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:55.088194 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:55.088230 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:55.091053 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:55.094306 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:55.094318 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:55.094337 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:55.095002 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:55.095425 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:55.095432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:55.095438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:55.095698 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:55.096391 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:55.096401 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:55.096891 (+ 490us) client_negotiation.cc:769] Sending connection context
0807 11:19:55.097217 (+ 326us) client_negotiation.cc:240] Negotiation successful
0807 11:19:55.097822 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":309,"threads_started":1}
I20260807 11:19:55.100649 17851 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:55.100965 17851 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:55.101684 17851 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:55.103788 17645 ts_manager.cc:194] Registered new tserver with Master: 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803)
I20260807 11:19:55.105330 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:43271
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:19:55.149626 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:55.485790 17857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:55.486106 17857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:55.486302 17857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:55.493490 17857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:55.493620 17857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:55.493737 17857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:19:55.503285 17857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:55.504484 17857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:55.505851 17857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:55.518898 17863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:55.518891 17864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:55.518831 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:55.519824 17857 server_base.cc:1034] running on GCE node
I20260807 11:19:55.520447 17857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:55.521332 17857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:55.522583 17857 hybrid_clock.cc:648] HybridClock initialized: now 1786101595522538 us; error 33 us; skew 500 ppm
I20260807 11:19:55.522889 17857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:55 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:55.525844 17857 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:19:55.527354 17857 webserver.cc:460] Webserver started at http://127.2.223.130:40451/ using document root <none> and password file <none>
I20260807 11:19:55.528039 17857 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:55.528182 17857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:55.528491 17857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:55.531332 17857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
format_stamp: "Formatted at 2026-08-07 11:19:55 on dist-test-slave-5l3k"
server_key: "3da562781686adc2d55809f113e48da8"
server_key_iv: "595cb4c11d14d9ca8074c14344aef86b"
server_key_version: "encryptionkey@0"
I20260807 11:19:55.532078 17857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
format_stamp: "Formatted at 2026-08-07 11:19:55 on dist-test-slave-5l3k"
server_key: "3da562781686adc2d55809f113e48da8"
server_key_iv: "595cb4c11d14d9ca8074c14344aef86b"
server_key_version: "encryptionkey@0"
I20260807 11:19:55.537104 17857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:19:55.540335 17875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.541651 17857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:19:55.541821 17857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
format_stamp: "Formatted at 2026-08-07 11:19:55 on dist-test-slave-5l3k"
server_key: "3da562781686adc2d55809f113e48da8"
server_key_iv: "595cb4c11d14d9ca8074c14344aef86b"
server_key_version: "encryptionkey@0"
I20260807 11:19:55.542006 17857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:55.562175 17857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:55.566305 17857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:55.566669 17857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:55.567718 17857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:55.569087 17857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:55.569190 17857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.569343 17857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:55.569427 17857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:55.602835 17857 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:40865
I20260807 11:19:55.602869 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:40865 every 8 connection(s)
I20260807 11:19:55.604120 17857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:19:55.608484 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17857
I20260807 11:19:55.608680 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:19:55.609175 2942 external_mini_cluster.cc:1428] Setting key 178f48523cac87e8ff7223db39cea782
Aug 07 11:19:55 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:55.626158 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:55.606825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:50839 (local address 127.2.223.190:39045)
0807 11:19:55.607314 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:55.607319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:55.608343 (+ 1024us) server_negotiation.cc:407] Connection header received
0807 11:19:55.610592 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:55.610596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:55.610668 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:55.610764 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:55.612477 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.613235 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.614833 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.615293 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.615429 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:55.619576 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:55.619582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:55.619585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:55.619615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:55.622325 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:55.623373 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:55.623377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:55.623379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:55.623508 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:55.624056 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:55.624060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:55.624062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:55.624321 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:55.624433 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:55.625405 (+ 972us) server_negotiation.cc:299] Negotiation successful
0807 11:19:55.625705 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
I20260807 11:19:55.626760 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:55.607430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:50839)
0807 11:19:55.608173 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:55.608214 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:55.610258 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:55.611106 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:55.611117 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:55.611725 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:55.612299 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:55.612316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.613396 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.613407 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:55.614685 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:55.614698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:55.615547 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:55.615555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:55.615776 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:55.616471 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:55.616498 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:55.619323 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:55.622510 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:55.622522 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:55.622541 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:55.623214 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:55.623636 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:55.623643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:55.623649 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:55.623917 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:55.624454 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:55.624465 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:55.624950 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 11:19:55.625205 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 11:19:55.625594 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":277,"threads_started":1}
I20260807 11:19:55.628228 17993 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:55.628558 17993 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:55.629467 17993 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:55.631098 17645 ts_manager.cc:194] Registered new tserver with Master: 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865)
I20260807 11:19:55.631903 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:50839
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:19:55.679332 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:56.017799 17999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:56.018117 17999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:56.018287 17999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:56.025553 17999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:56.025699 17999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:56.025893 17999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:19:56.035509 17999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:56.036790 17999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:56.038118 17999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:56.051491 18005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:56.051687 18004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:56.051764 18007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:56.052711 17999 server_base.cc:1034] running on GCE node
I20260807 11:19:56.053314 17999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:56.054235 17999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:56.055464 17999 hybrid_clock.cc:648] HybridClock initialized: now 1786101596055382 us; error 65 us; skew 500 ppm
I20260807 11:19:56.055822 17999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:56 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:56.059021 17999 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:19:56.060465 17999 webserver.cc:460] Webserver started at http://127.2.223.131:44483/ using document root <none> and password file <none>
I20260807 11:19:56.061180 17999 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:56.061316 17999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:56.061693 17999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:19:56.064867 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "3bd517cc9dba482384feb361d28126fe"
format_stamp: "Formatted at 2026-08-07 11:19:56 on dist-test-slave-5l3k"
server_key: "50aa1c3ab5bf9ec51450fcd7326ad2a4"
server_key_iv: "d1ed53d8c39c3db15aa2ae50e08f90ff"
server_key_version: "encryptionkey@0"
I20260807 11:19:56.065685 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "3bd517cc9dba482384feb361d28126fe"
format_stamp: "Formatted at 2026-08-07 11:19:56 on dist-test-slave-5l3k"
server_key: "50aa1c3ab5bf9ec51450fcd7326ad2a4"
server_key_iv: "d1ed53d8c39c3db15aa2ae50e08f90ff"
server_key_version: "encryptionkey@0"
I20260807 11:19:56.070936 17999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 11:19:56.074575 18015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:56.076143 17999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:19:56.076361 17999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "3bd517cc9dba482384feb361d28126fe"
format_stamp: "Formatted at 2026-08-07 11:19:56 on dist-test-slave-5l3k"
server_key: "50aa1c3ab5bf9ec51450fcd7326ad2a4"
server_key_iv: "d1ed53d8c39c3db15aa2ae50e08f90ff"
server_key_version: "encryptionkey@0"
I20260807 11:19:56.076587 17999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:56.109193 17851 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:56.114004 17999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:56.118207 17999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:56.118544 17999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:56.119542 17999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:56.120828 17999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:56.120910 17999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:56.121006 17999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:56.121052 17999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:56.155606 17999 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:39611
I20260807 11:19:56.155728 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:39611 every 8 connection(s)
I20260807 11:19:56.156960 17999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:19:56.159665 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 17999
I20260807 11:19:56.159878 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:19:56.160447 2942 external_mini_cluster.cc:1428] Setting key 7a8036109f95b4ef3e7ad6fd1840f88e
Aug 07 11:19:56 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:56.177568 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.159701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:56145 (local address 127.2.223.190:39045)
0807 11:19:56.160211 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.160217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.161362 (+ 1145us) server_negotiation.cc:407] Connection header received
0807 11:19:56.163728 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.163734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.163827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.163921 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:56.165343 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.166124 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.167802 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.168203 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.168304 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.171796 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:56.171801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:56.171805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:56.171860 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:56.174565 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.175340 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.175346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.175350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.175489 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.176015 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.176019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.176022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.176375 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:56.176513 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.177136 (+ 623us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.177297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
I20260807 11:19:56.178628 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.160332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:56145)
0807 11:19:56.161162 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.161206 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.163458 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.164140 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.164149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.164648 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:56.165201 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.165217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.166296 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.166300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.167664 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.167677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.168398 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.168402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.168596 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.169226 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:56.169253 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:56.171613 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:56.174703 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:56.174709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:56.174721 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:56.175216 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:56.175623 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:56.175627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:56.175629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:56.175867 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:56.176509 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:56.176515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:56.176818 (+ 303us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.177045 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.177421 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":324,"threads_started":1}
I20260807 11:19:56.180050 18138 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:56.180365 18138 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:56.181131 18138 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:56.182579 17645 ts_manager.cc:194] Registered new tserver with Master: 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611)
I20260807 11:19:56.183452 17645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:56145
I20260807 11:19:56.188347 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:19:56.205317 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.193058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.2.223.190:39045)
0807 11:19:56.193299 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.193308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.193757 (+ 449us) server_negotiation.cc:407] Connection header received
0807 11:19:56.193897 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.193900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.193963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.194041 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:56.195466 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.196142 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.197503 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.197928 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.198020 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.200500 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:56.200505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:56.200508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:56.200535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:56.202942 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.203603 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.203606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.203608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.203719 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.204033 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.204036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.204037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.204323 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:56.204491 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.204779 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.204937 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:19:56.209360 17645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:19:56.216347 17645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:19:56.239676 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.229551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.2.223.129:34803)
0807 11:19:56.230290 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.230302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.231150 (+ 848us) server_negotiation.cc:407] Connection header received
0807 11:19:56.231256 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.231266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.231501 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.231692 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:56.232541 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.234042 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.236359 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.237833 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.237977 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.238251 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.239171 (+ 920us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.239342 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":1474,"threads_started":1}
I20260807 11:19:56.239750 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.228964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:34803 (local address 127.0.0.1:50560)
0807 11:19:56.230875 (+ 1911us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.230896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.231013 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.231945 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.231951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.231983 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:56.232357 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.232369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.234207 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.234213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.236189 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.236202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.238418 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.238428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.238775 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.238941 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.239078 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.239298 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1770,"mutex_wait_us":146,"thread_start_us":773,"threads_started":1}
I20260807 11:19:56.239849 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.228905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.2.223.131:39611)
0807 11:19:56.229578 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.229590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.230168 (+ 578us) server_negotiation.cc:407] Connection header received
0807 11:19:56.230497 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.230509 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.230791 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.230995 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:56.232138 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.233699 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.236614 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.238002 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.238113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.238317 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.239316 (+ 999us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.239492 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
I20260807 11:19:56.240024 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.231369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.2.223.130:40865)
0807 11:19:56.232336 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.232342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.232364 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:19:56.232434 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.232439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.232680 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.232913 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:56.233488 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.235076 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.237268 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.238619 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.238830 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.239039 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.239513 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.239697 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":688,"threads_started":1}
I20260807 11:19:56.240468 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.230876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40865 (local address 127.0.0.1:54532)
0807 11:19:56.231680 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.231697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.231818 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.232920 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.232925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.232948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:56.233307 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.233318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.235267 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.235273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.237115 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.237126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.238890 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.238896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.239091 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.239125 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.239215 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.239340 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":3,"thread_start_us":700,"threads_started":1}
I20260807 11:19:56.241322 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.228715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:39611 (local address 127.0.0.1:36958)
0807 11:19:56.229996 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.230045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.230260 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.231334 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.231345 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.231373 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:56.231902 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.231919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.233836 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.233842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.236450 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.236465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.238455 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.238460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.238832 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.238941 (+ 109us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.239078 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.240258 (+ 1180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":1035,"thread_start_us":1839,"threads_started":1}
I20260807 11:19:56.244004 17923 tablet_service.cc:1467] Processing CreateTablet for tablet b494126b071b44038c0b03ee92696ff9 (DEFAULT_TABLE table=test-table [id=7351c99581384f72967f1f78bd61a390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:56.244223 17781 tablet_service.cc:1467] Processing CreateTablet for tablet b494126b071b44038c0b03ee92696ff9 (DEFAULT_TABLE table=test-table [id=7351c99581384f72967f1f78bd61a390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:56.244807 18066 tablet_service.cc:1467] Processing CreateTablet for tablet b494126b071b44038c0b03ee92696ff9 (DEFAULT_TABLE table=test-table [id=7351c99581384f72967f1f78bd61a390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:56.245440 17923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b494126b071b44038c0b03ee92696ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:56.245715 17781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b494126b071b44038c0b03ee92696ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:56.246218 18066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b494126b071b44038c0b03ee92696ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:56.257287 18158 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Bootstrap starting.
I20260807 11:19:56.260885 18158 tablet_bootstrap.cc:654] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:56.262115 18158 log.cc:824] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:56.263845 18159 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Bootstrap starting.
I20260807 11:19:56.264781 18160 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Bootstrap starting.
I20260807 11:19:56.267777 18159 tablet_bootstrap.cc:654] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:56.268096 18158 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: No bootstrap required, opened a new log
I20260807 11:19:56.268378 18158 ts_tablet_manager.cc:1397] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260807 11:19:56.268369 18160 tablet_bootstrap.cc:654] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:56.269193 18159 log.cc:824] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:56.269654 18160 log.cc:824] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:56.273175 18160 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: No bootstrap required, opened a new log
I20260807 11:19:56.273216 18159 tablet_bootstrap.cc:492] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: No bootstrap required, opened a new log
I20260807 11:19:56.273449 18160 ts_tablet_manager.cc:1397] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 11:19:56.273475 18159 ts_tablet_manager.cc:1397] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260807 11:19:56.279381 18158 raft_consensus.cc:348] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.279894 18158 raft_consensus.cc:374] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:56.280015 18158 raft_consensus.cc:729] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ac3de4ab214f0e96ad6ec4aa922eb5, State: Initialized, Role: FOLLOWER
I20260807 11:19:56.280783 18158 consensus_queue.cc:260] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.281083 18160 raft_consensus.cc:348] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.281620 18160 raft_consensus.cc:374] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:56.281781 18160 raft_consensus.cc:729] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd517cc9dba482384feb361d28126fe, State: Initialized, Role: FOLLOWER
I20260807 11:19:56.282194 17993 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:56.282951 18158 ts_tablet_manager.cc:1428] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260807 11:19:56.283080 18160 consensus_queue.cc:260] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.284307 18138 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:56.284430 18159 raft_consensus.cc:348] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.284823 18159 raft_consensus.cc:374] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:56.284925 18160 ts_tablet_manager.cc:1428] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260807 11:19:56.284943 18159 raft_consensus.cc:729] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9ddca38ca84f1b9f765297cf63cf9b, State: Initialized, Role: FOLLOWER
I20260807 11:19:56.285542 18159 consensus_queue.cc:260] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.287313 18159 ts_tablet_manager.cc:1428] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260807 11:19:56.328578 17852 tablet.cc:2369] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:19:56.356186 17994 tablet.cc:2369] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:19:56.409034 18139 tablet.cc:2369] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:56.454295 18166 raft_consensus.cc:482] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:56.454633 18166 raft_consensus.cc:504] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.456243 18166 leader_election.cc:288] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803), 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865)
I20260807 11:19:56.463181 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.456602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:34803 (local address 127.2.223.131:57989)
0807 11:19:56.456858 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.456880 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.457046 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.457635 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.457640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.457683 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:56.457935 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.457949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.459255 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.459259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.461112 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.461127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.462563 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.462567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.462709 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.462733 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.462884 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.462980 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 11:19:56.463327 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.456774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:57989 (local address 127.2.223.129:34803)
0807 11:19:56.457018 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.457024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.457048 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:19:56.457215 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.457221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.457308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.457418 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:56.458108 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.459127 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.461273 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.462380 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.462608 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.462687 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.462923 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.463069 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:19:56.464031 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.457303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:59603 (local address 127.2.223.130:40865)
0807 11:19:56.457519 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.457527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.457971 (+ 444us) server_negotiation.cc:407] Connection header received
0807 11:19:56.458136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.458141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.458222 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.458319 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:56.458858 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.459671 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.461805 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.463153 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.463295 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.463380 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.463682 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.463830 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:19:56.464126 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.457067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40865 (local address 127.2.223.131:59603)
0807 11:19:56.457858 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:56.457879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:56.458001 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:56.458338 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:56.458341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:56.458360 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:56.458712 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.458720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.459821 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.459826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.461628 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:56.461638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.463355 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.463361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:56.463518 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.463552 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:19:56.463646 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 11:19:56.463764 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":439,"threads_started":1}
I20260807 11:19:56.464632 17803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b494126b071b44038c0b03ee92696ff9" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" is_pre_election: true
I20260807 11:19:56.465234 17803 raft_consensus.cc:2454] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 0.
I20260807 11:19:56.465305 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b494126b071b44038c0b03ee92696ff9" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" is_pre_election: true
I20260807 11:19:56.465826 17945 raft_consensus.cc:2454] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 0.
I20260807 11:19:56.465986 18016 leader_election.cc:302] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd517cc9dba482384feb361d28126fe, 3d9ddca38ca84f1b9f765297cf63cf9b; no voters:
I20260807 11:19:56.466415 18166 raft_consensus.cc:2781] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:19:56.466610 18166 raft_consensus.cc:482] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:19:56.466784 18166 raft_consensus.cc:3037] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:56.469085 18166 raft_consensus.cc:504] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.469620 18166 leader_election.cc:288] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 election: Requested vote from peers 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803), 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865)
I20260807 11:19:56.470072 17803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b494126b071b44038c0b03ee92696ff9" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
I20260807 11:19:56.470101 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b494126b071b44038c0b03ee92696ff9" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
I20260807 11:19:56.470274 17803 raft_consensus.cc:3037] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:56.470283 17945 raft_consensus.cc:3037] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:56.472625 17803 raft_consensus.cc:2454] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 1.
I20260807 11:19:56.472638 17945 raft_consensus.cc:2454] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 1.
I20260807 11:19:56.473030 18016 leader_election.cc:302] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd517cc9dba482384feb361d28126fe, 58ac3de4ab214f0e96ad6ec4aa922eb5; no voters:
I20260807 11:19:56.473368 18166 raft_consensus.cc:2781] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:56.473834 18166 raft_consensus.cc:686] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 1 LEADER]: Becoming Leader. State: Replica: 3bd517cc9dba482384feb361d28126fe, State: Running, Role: LEADER
I20260807 11:19:56.474294 18166 consensus_queue.cc:237] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } }
I20260807 11:19:56.479116 17644 catalog_manager.cc:5495] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd517cc9dba482384feb361d28126fe (127.2.223.131). New cstate: current_term: 1 leader_uuid: "3bd517cc9dba482384feb361d28126fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } health_report { overall_health: UNKNOWN } } }
I20260807 11:19:56.537807 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.531536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.2.223.131:39611)
0807 11:19:56.531809 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.531817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.531841 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:19:56.532138 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.532142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.532228 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.532417 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:19:56.532846 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.533667 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.535312 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.535781 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.535943 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.536531 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:19:56.537079 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:19:56.537284 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.537356 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.537457 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:19:56.546459 17945 raft_consensus.cc:1261] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Refusing update from remote peer 3bd517cc9dba482384feb361d28126fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:56.546458 17803 raft_consensus.cc:1261] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Refusing update from remote peer 3bd517cc9dba482384feb361d28126fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:56.547574 18169 consensus_queue.cc:1035] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:56.547999 18166 consensus_queue.cc:1035] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:56.557489 18172 mvcc.cc:205] Tried to move back new op lower bound from 7315872139443638272 to 7315872139162423296. Current Snapshot: MvccSnapshot[applied={T|T < 7315872139443638272}]
I20260807 11:19:56.560506 18174 mvcc.cc:205] Tried to move back new op lower bound from 7315872139443638272 to 7315872139162423296. Current Snapshot: MvccSnapshot[applied={T|T < 7315872139443638272}]
I20260807 11:19:56.571908 17645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:56.572208 17645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:56.575634 17645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Sending DeleteTablet for 3 replicas of tablet b494126b071b44038c0b03ee92696ff9
I20260807 11:19:56.577029 17781 tablet_service.cc:1514] Processing DeleteTablet for tablet b494126b071b44038c0b03ee92696ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:56 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:50560
I20260807 11:19:56.577123 18066 tablet_service.cc:1514] Processing DeleteTablet for tablet b494126b071b44038c0b03ee92696ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:56 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:36958
I20260807 11:19:56.577123 17923 tablet_service.cc:1514] Processing DeleteTablet for tablet b494126b071b44038c0b03ee92696ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:56 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:54532
I20260807 11:19:56.578178 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17711
I20260807 11:19:56.589736 18183 tablet_replica.cc:330] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: stopping tablet replica
I20260807 11:19:56.590421 18183 raft_consensus.cc:2229] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:19:56.591513 18183 raft_consensus.cc:2258] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:56.594410 18183 ts_tablet_manager.cc:1905] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:56.602986 18184 tablet_replica.cc:330] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: stopping tablet replica
I20260807 11:19:56.603390 18184 raft_consensus.cc:2229] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:56.603948 18184 raft_consensus.cc:2258] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:56.604499 18182 tablet_replica.cc:330] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: stopping tablet replica
I20260807 11:19:56.604897 18182 raft_consensus.cc:2229] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:56.605528 18182 raft_consensus.cc:2258] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:56.606746 18184 ts_tablet_manager.cc:1905] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:56.607426 18182 ts_tablet_manager.cc:1905] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:56.614442 18182 ts_tablet_manager.cc:1918] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:56.614672 18182 log.cc:1196] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/b494126b071b44038c0b03ee92696ff9
I20260807 11:19:56.615221 18182 ts_tablet_manager.cc:1939] T b494126b071b44038c0b03ee92696ff9 P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting consensus metadata
Aug 07 11:19:56 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:19:56.617149 17626 catalog_manager.cc:4841] TS 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803): tablet b494126b071b44038c0b03ee92696ff9 (table test-table [id=7351c99581384f72967f1f78bd61a390]) successfully deleted
I20260807 11:19:56.619642 18183 ts_tablet_manager.cc:1918] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:56.619872 18183 log.cc:1196] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/b494126b071b44038c0b03ee92696ff9
I20260807 11:19:56.620487 18183 ts_tablet_manager.cc:1939] T b494126b071b44038c0b03ee92696ff9 P 3bd517cc9dba482384feb361d28126fe: Deleting consensus metadata
I20260807 11:19:56.622215 18184 ts_tablet_manager.cc:1918] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:56.622311 17627 catalog_manager.cc:4841] TS 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611): tablet b494126b071b44038c0b03ee92696ff9 (table test-table [id=7351c99581384f72967f1f78bd61a390]) successfully deleted
I20260807 11:19:56.622422 18184 log.cc:1196] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/b494126b071b44038c0b03ee92696ff9
I20260807 11:19:56.622979 18184 ts_tablet_manager.cc:1939] T b494126b071b44038c0b03ee92696ff9 P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting consensus metadata
I20260807 11:19:56.624564 17626 catalog_manager.cc:4841] TS 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865): tablet b494126b071b44038c0b03ee92696ff9 (table test-table [id=7351c99581384f72967f1f78bd61a390]) successfully deleted
I20260807 11:19:56.627561 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.579634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.2.223.129:34803)
0807 11:19:56.580255 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.580262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.603828 (+ 23566us) server_negotiation.cc:407] Connection header received
0807 11:19:56.604032 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.604038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.604140 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.604283 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:56.607036 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.608222 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.610423 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.611066 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.611200 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.616838 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:56.616853 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:56.616889 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:56.616936 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:56.621006 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.625140 (+ 4134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.625150 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.625153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.625337 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.625780 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.625786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.625791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.626272 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:56.626613 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.626930 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.627178 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490}
I20260807 11:19:56.628400 17846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:56.720255 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17711
I20260807 11:19:56.742679 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17857
Aug 07 11:19:56 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:19:56.760450 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.743544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.2.223.130:40865)
0807 11:19:56.743722 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.743726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.743745 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:19:56.743943 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.743947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.744056 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.744142 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:56.745922 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.746919 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.748882 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.749357 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.749468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.754577 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:56.754599 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:56.754623 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:56.754669 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:56.757565 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.758263 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.758270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.758273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.758396 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.758826 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.758832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.758835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.759243 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:56.759567 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.759871 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.760106 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:19:56.761181 17988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:56.854472 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17857
I20260807 11:19:56.877981 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17999
Aug 07 11:19:56 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:19:56.897456 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:56.879213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.2.223.131:39611)
0807 11:19:56.879444 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:56.879452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:56.879497 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:19:56.879596 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:56.879600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:56.879688 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:56.879846 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:56.881608 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.882744 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.884329 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:56.884883 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:56.885068 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:56.890994 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:56.891007 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:56.891027 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:56.891067 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:56.894302 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.895250 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.895257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.895259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.895426 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:56.895871 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:56.895875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:56.895877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:56.896255 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:56.896519 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:56.896901 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 11:19:56.897078 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:19:56.898305 18132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:57.025437 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17999
I20260807 11:19:57.050606 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 17607
I20260807 11:19:57.051745 17679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:57.164578 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 17607
I20260807 11:19:57.191197 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--webserver_interface=127.2.223.190
--webserver_port=43495
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:19:57.543558 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:57.543901 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:57.544077 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:57.551239 18193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:19:57.551388 18193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:57.551502 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:57.551609 18193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:19:57.551702 18193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:19:57.561560 18193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:57.562958 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:57.564328 18193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:57.590929 18199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:57.591019 18200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:57.590987 18202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:57.591915 18193 server_base.cc:1034] running on GCE node
I20260807 11:19:57.592799 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:57.594554 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:57.595798 18193 hybrid_clock.cc:648] HybridClock initialized: now 1786101597595747 us; error 55 us; skew 500 ppm
I20260807 11:19:57.596217 18193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:57 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:57.614056 18193 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:19:57.616015 18193 webserver.cc:460] Webserver started at http://127.2.223.190:43495/ using document root <none> and password file <none>
I20260807 11:19:57.616860 18193 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:57.617043 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:57.623354 18193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 11:19:57.626439 18210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:57.627897 18193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:19:57.628155 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "e7d58cb3825c4dd388e08a583385853b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "6387abecb9530524fa223e38ecfb9b73"
server_key_iv: "79a56fa2d54ddc8bee66bf67c9e1f8d1"
server_key_version: "encryptionkey@0"
I20260807 11:19:57.629302 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:57.648751 18193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:57.653201 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:57.653501 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:57.673319 18193 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:39045
I20260807 11:19:57.673338 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:39045 every 8 connection(s)
I20260807 11:19:57.674919 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:19:57.681979 18266 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:19:57.682530 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18193
I20260807 11:19:57.683422 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:34803
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=38629
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:19:57.685474 18266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap starting.
I20260807 11:19:57.708928 18266 log.cc:824] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:57.721179 18266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:19:57.721747 18266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap complete.
I20260807 11:19:57.731354 18266 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:57.732223 18266 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Initialized, Role: FOLLOWER
I20260807 11:19:57.732983 18266 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:57.733253 18266 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:19:57.733383 18266 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:19:57.733593 18266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:19:57.737567 18266 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:57.738193 18266 leader_election.cc:302] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d58cb3825c4dd388e08a583385853b; no voters:
I20260807 11:19:57.738929 18266 leader_election.cc:288] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:19:57.739095 18271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:19:57.740583 18271 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 LEADER]: Becoming Leader. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Running, Role: LEADER
I20260807 11:19:57.741223 18271 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:19:57.741405 18266 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:19:57.744196 18272 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:19:57.744675 18272 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:57.747080 18276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:19:57.747014 18273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d58cb3825c4dd388e08a583385853b. Latest consensus state: current_term: 2 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:19:57.747612 18273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:19:57.756788 18276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:19:57.758258 18276 catalog_manager.cc:1259] Loaded cluster ID: 65df56d7ffc0460f93828cbe91333d26
I20260807 11:19:57.758411 18276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:19:57.762048 18276 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 11:19:57.788540 18276 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 11:19:57.814983 18276 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:19:57.817130 18276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Loaded TSK: 0
I20260807 11:19:57.818764 18276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:19:58.059665 18268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:58.059947 18268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:58.060094 18268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:58.067651 18268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:58.067759 18268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:58.067955 18268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:19:58.077517 18268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=38629
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:58.078905 18268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:58.080231 18268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:58.093868 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:58.094035 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:58.094839 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:58.095409 18268 server_base.cc:1034] running on GCE node
I20260807 11:19:58.096163 18268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:58.097474 18268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:58.098711 18268 hybrid_clock.cc:648] HybridClock initialized: now 1786101598098643 us; error 48 us; skew 500 ppm
I20260807 11:19:58.099061 18268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:58 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:58.102818 18268 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:19:58.104497 18268 webserver.cc:460] Webserver started at http://127.2.223.129:38629/ using document root <none> and password file <none>
I20260807 11:19:58.105373 18268 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:58.105531 18268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:58.112370 18268 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:19:58.116010 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.117750 18268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260807 11:19:58.117918 18268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "c5680442316ae3c88a390db730f3a68e"
server_key_iv: "aca7b3ffb55fcda2a43f48b21ab6092f"
server_key_version: "encryptionkey@0"
I20260807 11:19:58.119338 18268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:58.144761 18268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:58.149932 18268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:58.150225 18268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:58.151458 18268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:58.153154 18268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:58.153259 18268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.153365 18268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:58.153435 18268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.189595 18268 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:34803
I20260807 11:19:58.189687 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:34803 every 8 connection(s)
I20260807 11:19:58.191279 18268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:19:58.196070 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18268
I20260807 11:19:58.196703 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:40865
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=40451
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:58 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:58.217168 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:58.194612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:42095)
0807 11:19:58.195428 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:58.195473 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:58.197365 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:58.199080 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:58.199088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:58.199589 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:58.200124 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:58.200142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.202587 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.202592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:58.203793 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:58.203806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.205242 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.205247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:58.205443 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:58.206169 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:58.206194 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:58.208479 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:58.212907 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:58.212914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:58.212926 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:58.213353 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:58.213895 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:58.213898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:58.213901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:58.214078 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:58.215149 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:58.215155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:58.215454 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 11:19:58.215673 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 11:19:58.216079 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":360,"threads_started":1}
I20260807 11:19:58.217790 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:58.194408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:42095 (local address 127.2.223.190:39045)
0807 11:19:58.195111 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:58.195123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:58.195639 (+ 516us) server_negotiation.cc:407] Connection header received
0807 11:19:58.197999 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:58.198021 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:58.198516 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:58.198946 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:58.200372 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.202360 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.203946 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.205034 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.205225 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:58.208778 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:58.208790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:58.208812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:58.208871 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:58.212753 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:58.213521 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:58.213533 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:58.213543 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:58.213780 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:58.214204 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:58.214211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:58.214217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:58.215005 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:58.215220 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:58.215982 (+ 762us) server_negotiation.cc:299] Negotiation successful
0807 11:19:58.216449 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":313,"threads_started":1}
I20260807 11:19:58.220569 18429 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:58.220901 18429 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:58.221653 18429 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:58.223927 18229 ts_manager.cc:194] Registered new tserver with Master: 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803)
I20260807 11:19:58.225605 18229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:42095
W20260807 11:19:58.541333 18433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:58.541679 18433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:58.541853 18433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:58.549295 18433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:58.549454 18433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:58.549651 18433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:19:58.559875 18433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:40865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=40451
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:58.561357 18433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:58.562949 18433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:58.576608 18438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:58.576735 18439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:58.576886 18441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:58.578059 18433 server_base.cc:1034] running on GCE node
I20260807 11:19:58.578802 18433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:58.579809 18433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:58.581022 18433 hybrid_clock.cc:648] HybridClock initialized: now 1786101598580972 us; error 33 us; skew 500 ppm
I20260807 11:19:58.581405 18433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:58 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:58.584556 18433 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:19:58.585968 18433 webserver.cc:460] Webserver started at http://127.2.223.130:40451/ using document root <none> and password file <none>
I20260807 11:19:58.586758 18433 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:58.586879 18433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:58.592614 18433 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:19:58.595419 18448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.596815 18433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:19:58.596957 18433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
format_stamp: "Formatted at 2026-08-07 11:19:55 on dist-test-slave-5l3k"
server_key: "3da562781686adc2d55809f113e48da8"
server_key_iv: "595cb4c11d14d9ca8074c14344aef86b"
server_key_version: "encryptionkey@0"
I20260807 11:19:58.597955 18433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:58.625165 18433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:58.629598 18433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:58.629884 18433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:58.630970 18433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:58.632499 18433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:58.632601 18433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.632699 18433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:58.632755 18433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:58.670744 18433 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:40865
I20260807 11:19:58.670802 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:40865 every 8 connection(s)
I20260807 11:19:58.672479 18433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:19:58.682435 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18433
I20260807 11:19:58.683322 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:39611
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=44483
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:19:58 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:58.698573 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:58.675302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:38041 (local address 127.2.223.190:39045)
0807 11:19:58.675529 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:58.675536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:58.676924 (+ 1388us) server_negotiation.cc:407] Connection header received
0807 11:19:58.678909 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:58.678916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:58.679049 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:58.679168 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:58.680625 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.681769 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.684370 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.685139 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.685276 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:58.690350 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:58.690359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:58.690367 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:58.690419 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:58.694358 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:58.695405 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:58.695413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:58.695419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:58.695603 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:58.696141 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:58.696149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:58.696154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:58.696562 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:58.696730 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:58.697833 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0807 11:19:58.698102 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:19:58.699406 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:58.675778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:38041)
0807 11:19:58.676726 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:58.676778 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:58.678630 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:58.679388 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:58.679396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:58.679889 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:58.680454 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:58.680471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.682001 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.682007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:58.684190 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:58.684210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:58.685398 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:58.685409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:58.685730 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:58.686611 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:58.686645 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:58.690009 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:58.694562 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:58.694574 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:58.694595 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:58.695249 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:58.695727 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:58.695734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:58.695740 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:58.696000 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:58.696705 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:58.696715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:58.697379 (+ 664us) client_negotiation.cc:769] Sending connection context
0807 11:19:58.697641 (+ 262us) client_negotiation.cc:240] Negotiation successful
0807 11:19:58.698087 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":408,"threads_started":1}
I20260807 11:19:58.701124 18568 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:58.701475 18568 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:58.702379 18568 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:58.704057 18229 ts_manager.cc:194] Registered new tserver with Master: 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865)
I20260807 11:19:58.705385 18229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:38041
W20260807 11:19:59.041065 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:19:59.041380 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:19:59.041582 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:19:59.048622 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:19:59.048770 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:19:59.048945 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:19:59.058435 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:39611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=44483
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:19:59.059824 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:19:59.061192 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:19:59.074669 18577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:59.074772 18578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:19:59.074800 18580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:19:59.075942 18571 server_base.cc:1034] running on GCE node
I20260807 11:19:59.076570 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:19:59.077539 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:19:59.078770 18571 hybrid_clock.cc:648] HybridClock initialized: now 1786101599078731 us; error 52 us; skew 500 ppm
I20260807 11:19:59.079123 18571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:19:59 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:19:59.082252 18571 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:19:59.083873 18571 webserver.cc:460] Webserver started at http://127.2.223.131:44483/ using document root <none> and password file <none>
I20260807 11:19:59.084528 18571 fs_manager.cc:359] Metadata directory not provided
I20260807 11:19:59.084646 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:19:59.090197 18571 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:19:59.093062 18587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:59.094460 18571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 11:19:59.094722 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "3bd517cc9dba482384feb361d28126fe"
format_stamp: "Formatted at 2026-08-07 11:19:56 on dist-test-slave-5l3k"
server_key: "50aa1c3ab5bf9ec51450fcd7326ad2a4"
server_key_iv: "d1ed53d8c39c3db15aa2ae50e08f90ff"
server_key_version: "encryptionkey@0"
I20260807 11:19:59.096421 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:19:59.116752 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:19:59.120671 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:19:59.120947 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:19:59.121963 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:19:59.123342 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:19:59.123476 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:59.123606 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:19:59.123701 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:19:59.158579 18571 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:39611
I20260807 11:19:59.158707 18710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:39611 every 8 connection(s)
I20260807 11:19:59.160053 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:19:59.168958 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18571
Aug 07 11:19:59 dist-test-slave-5l3k krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:19:59.180768 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.162645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:37389 (local address 127.2.223.190:39045)
0807 11:19:59.162827 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.162832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.164063 (+ 1231us) server_negotiation.cc:407] Connection header received
0807 11:19:59.165935 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.165941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.166038 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.166152 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:59.167689 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.168420 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.170125 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.170932 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.171064 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.174688 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:59.174694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:59.174697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:59.174728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:59.177577 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.178339 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.178344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.178350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.178529 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.179000 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.179005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.179009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.179279 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:59.179457 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.180132 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.180396 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:19:59.181716 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.163186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:37389)
0807 11:19:59.163893 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.163932 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.165662 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.166394 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.166402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.166959 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:19:59.167502 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.167520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.168607 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.168611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.169983 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.169996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.171177 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.171186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.171469 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.172221 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:19:59.172249 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:19:59.174420 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:19:59.177754 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:59.177763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:59.177778 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:59.178203 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:59.178666 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:19:59.178673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:19:59.178679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:19:59.178874 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:19:59.179419 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:19:59.179426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:19:59.179760 (+ 334us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.180025 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.180565 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":285,"threads_started":1}
I20260807 11:19:59.183540 18711 heartbeater.cc:347] Connected to a master server at 127.2.223.190:39045
I20260807 11:19:59.183892 18711 heartbeater.cc:464] Registering TS with master...
I20260807 11:19:59.184829 18711 heartbeater.cc:511] Master 127.2.223.190:39045 requested a full tablet report, sending...
I20260807 11:19:59.184867 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.169941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.2.223.190:39045)
0807 11:19:59.170956 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.170960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.170976 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:19:59.171018 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.171022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.171075 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.171201 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:59.172671 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.173399 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.174969 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.175418 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.175567 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.178237 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:59.178242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:59.178245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:59.178272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:59.181234 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.182040 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.182047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.182051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.182203 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.182944 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.182950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.182954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.183360 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:59.183639 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.184100 (+ 461us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.184398 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":775,"threads_started":1}
I20260807 11:19:59.186427 18228 ts_manager.cc:194] Registered new tserver with Master: 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611)
I20260807 11:19:59.187697 18228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:37389
I20260807 11:19:59.188398 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:19:59.196066 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.189908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.2.223.190:39045)
0807 11:19:59.190301 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.190308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.190327 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:19:59.190419 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.190424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.190659 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.190815 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:19:59.191332 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.192363 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.193895 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.194530 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.194661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.195100 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:19:59.195574 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:19:59.195682 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.195750 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.195805 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260807 11:19:59.196993 18228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:19:59.205005 18228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:19:59.228631 18429 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:59.231604 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.218314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:39611 (local address 127.0.0.1:59896)
0807 11:19:59.222548 (+ 4234us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.222584 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.222739 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.223820 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.223825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.223855 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.224246 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.224261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.226423 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.226429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.228741 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.228754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.230605 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.230613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.230817 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.230873 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.231016 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.231202 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4049,"thread_start_us":3866,"threads_started":1}
I20260807 11:19:59.231612 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.218149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.2.223.131:39611)
0807 11:19:59.219276 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.219289 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.223250 (+ 3961us) server_negotiation.cc:407] Connection header received
0807 11:19:59.223354 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.223362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.223594 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.224443 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.224778 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.226284 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.228922 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.230369 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.230538 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.230786 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.231074 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.231240 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":502,"threads_started":1}
I20260807 11:19:59.232275 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.221731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:34803 (local address 127.0.0.1:40036)
0807 11:19:59.223335 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.223355 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.223507 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.224734 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.224739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.224761 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.225186 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.225202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.227358 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.227364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.229486 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.229499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.231472 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.231478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.231675 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.231713 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.231784 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.231927 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"thread_start_us":1270,"threads_started":1}
I20260807 11:19:59.232928 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.217523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40865 (local address 127.0.0.1:55824)
0807 11:19:59.221327 (+ 3804us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.221371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.221547 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.223230 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.223238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.223270 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.223738 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.223755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.225990 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.225996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.229698 (+ 3702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.229713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.232050 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.232056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.232236 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.232273 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.232375 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.232503 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3555,"mutex_wait_us":1,"thread_start_us":3391,"threads_started":1}
I20260807 11:19:59.233235 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.217719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.2.223.130:40865)
0807 11:19:59.222169 (+ 4450us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.222178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.222212 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:19:59.222313 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.222320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.222676 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.222910 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.223977 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.225849 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.229955 (+ 4106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.231390 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.231527 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.232093 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.232735 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.232914 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4244,"thread_start_us":4092,"threads_started":1}
I20260807 11:19:59.233850 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.221955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.2.223.129:34803)
0807 11:19:59.224108 (+ 2153us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.224137 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.224166 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:19:59.224256 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.224265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.224490 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.225410 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.225607 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.227179 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.229685 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.231206 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.232225 (+ 1019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.232831 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.232905 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.233258 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":1852,"threads_started":1}
I20260807 11:19:59.235790 18642 tablet_service.cc:1467] Processing CreateTablet for tablet 53a20535a60e44eb8ce71e6769115c1a (DEFAULT_TABLE table=test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:59.237020 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a20535a60e44eb8ce71e6769115c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:59.236819 18498 tablet_service.cc:1467] Processing CreateTablet for tablet 53a20535a60e44eb8ce71e6769115c1a (DEFAULT_TABLE table=test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:59.237448 18359 tablet_service.cc:1467] Processing CreateTablet for tablet 53a20535a60e44eb8ce71e6769115c1a (DEFAULT_TABLE table=test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:19:59.238257 18498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a20535a60e44eb8ce71e6769115c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:59.238848 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a20535a60e44eb8ce71e6769115c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:19:59.249442 18729 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Bootstrap starting.
I20260807 11:19:59.252576 18729 tablet_bootstrap.cc:654] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:59.253598 18730 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Bootstrap starting.
I20260807 11:19:59.253903 18729 log.cc:824] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:59.255134 18731 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Bootstrap starting.
I20260807 11:19:59.258108 18731 tablet_bootstrap.cc:654] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:59.258138 18730 tablet_bootstrap.cc:654] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Neither blocks nor log segments found. Creating new log.
I20260807 11:19:59.259250 18730 log.cc:824] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:59.259455 18731 log.cc:824] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:19:59.261243 18729 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: No bootstrap required, opened a new log
I20260807 11:19:59.261545 18729 ts_tablet_manager.cc:1397] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260807 11:19:59.262317 18730 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: No bootstrap required, opened a new log
I20260807 11:19:59.262605 18731 tablet_bootstrap.cc:492] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: No bootstrap required, opened a new log
I20260807 11:19:59.262612 18730 ts_tablet_manager.cc:1397] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260807 11:19:59.262887 18731 ts_tablet_manager.cc:1397] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260807 11:19:59.269869 18729 raft_consensus.cc:348] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.270267 18729 raft_consensus.cc:374] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:59.270380 18729 raft_consensus.cc:729] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd517cc9dba482384feb361d28126fe, State: Initialized, Role: FOLLOWER
I20260807 11:19:59.271051 18729 consensus_queue.cc:260] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.272212 18711 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:59.272946 18729 ts_tablet_manager.cc:1428] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 11:19:59.273439 18730 raft_consensus.cc:348] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.273439 18731 raft_consensus.cc:348] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.273867 18730 raft_consensus.cc:374] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:59.273867 18731 raft_consensus.cc:374] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:19:59.273974 18731 raft_consensus.cc:729] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9ddca38ca84f1b9f765297cf63cf9b, State: Initialized, Role: FOLLOWER
I20260807 11:19:59.273974 18730 raft_consensus.cc:729] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ac3de4ab214f0e96ad6ec4aa922eb5, State: Initialized, Role: FOLLOWER
I20260807 11:19:59.274591 18731 consensus_queue.cc:260] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.274591 18730 consensus_queue.cc:260] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.276297 18568 heartbeater.cc:503] Master 127.2.223.190:39045 was elected leader, sending a full tablet report...
I20260807 11:19:59.276497 18731 ts_tablet_manager.cc:1428] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260807 11:19:59.276890 18730 ts_tablet_manager.cc:1428] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260807 11:19:59.411686 18712 tablet.cc:2369] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:19:59.424685 18569 tablet.cc:2369] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:19:59.443743 18430 tablet.cc:2369] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:19:59.565160 18735 raft_consensus.cc:482] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:59.565466 18735 raft_consensus.cc:504] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.567392 18737 raft_consensus.cc:482] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:59.567476 18735 leader_election.cc:288] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865), 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803)
I20260807 11:19:59.567749 18737 raft_consensus.cc:504] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.576848 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.567854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40865 (local address 127.2.223.131:59553)
0807 11:19:59.568106 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.568128 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.568272 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.568826 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.568831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.568862 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.569121 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.569137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.570746 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.570751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.573309 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.573330 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.575790 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.575798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.576093 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.576145 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.576256 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.576449 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 11:19:59.577749 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.567954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:59553 (local address 127.2.223.130:40865)
0807 11:19:59.568161 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.568168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.568192 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:19:59.568452 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.568459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.568547 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.568639 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.569331 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.570533 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.573560 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.575381 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.575564 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.575675 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.577172 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.577369 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:19:59.577907 18738 raft_consensus.cc:482] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:19:59.578226 18738 raft_consensus.cc:504] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.579008 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.568689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:48051 (local address 127.2.223.129:34803)
0807 11:19:59.568901 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.568908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.569601 (+ 693us) server_negotiation.cc:407] Connection header received
0807 11:19:59.569766 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.569770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.569862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.569964 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.570930 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.572312 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.575492 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.577245 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.577408 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.577515 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.578412 (+ 897us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.578641 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:19:59.580456 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.568317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:34803 (local address 127.2.223.131:48051)
0807 11:19:59.569448 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.569465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.569615 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.570149 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.570155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.570196 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.570679 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.570701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.572584 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.572592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.575269 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.575289 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.577989 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.577998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.578201 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.578233 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.579784 (+ 1551us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.579994 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":763,"threads_started":1}
I20260807 11:19:59.580364 18737 leader_election.cc:288] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611), 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803)
I20260807 11:19:59.580606 18738 leader_election.cc:288] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865), 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611)
I20260807 11:19:59.586959 18518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" is_pre_election: true
I20260807 11:19:59.587483 18518 raft_consensus.cc:2454] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 0.
I20260807 11:19:59.588228 18588 leader_election.cc:302] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd517cc9dba482384feb361d28126fe, 58ac3de4ab214f0e96ad6ec4aa922eb5; no voters:
I20260807 11:19:59.588726 18735 raft_consensus.cc:2781] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:19:59.588900 18735 raft_consensus.cc:482] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:19:59.589028 18735 raft_consensus.cc:3037] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:59.589762 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.570353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:34803 (local address 127.2.223.130:37121)
0807 11:19:59.578502 (+ 8149us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.578524 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.578660 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.580701 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.580709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.580752 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.581646 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.581675 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.584087 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.584095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.586756 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.586775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.588914 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.588920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.589091 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.589135 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.589247 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.589376 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7968,"thread_start_us":15054,"threads_started":1}
I20260807 11:19:59.590435 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.570594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:37121 (local address 127.2.223.129:34803)
0807 11:19:59.579287 (+ 8693us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.579295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.579318 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:19:59.579387 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.579394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.579493 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.579610 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.581880 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.583883 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.586969 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.588456 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.588647 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.588737 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.589970 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.590110 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8557,"threads_started":1}
I20260807 11:19:59.593526 18735 raft_consensus.cc:504] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.594820 18518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
I20260807 11:19:59.595106 18518 raft_consensus.cc:3037] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:59.595594 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.585729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:58601 (local address 127.2.223.130:40865)
0807 11:19:59.585870 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.585875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.585891 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:19:59.585940 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.585945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.586016 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.586115 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.588658 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.589936 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.592895 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.594182 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.594331 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.594394 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.595140 (+ 746us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.595299 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 11:19:59.596230 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.581278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:40865 (local address 127.2.223.129:58601)
0807 11:19:59.583298 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.583321 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.583838 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.588000 (+ 4162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.588006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.588037 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.588496 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.588507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.590258 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.590263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.592702 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.592720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.594721 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.594731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.594924 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.594962 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.595738 (+ 776us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.595876 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1756,"thread_start_us":10121,"threads_started":1}
I20260807 11:19:59.596987 18735 leader_election.cc:288] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 election: Requested vote from peers 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865), 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803)
I20260807 11:19:59.597234 18517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" is_pre_election: true
I20260807 11:19:59.598038 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" is_pre_election: true
I20260807 11:19:59.598554 18380 raft_consensus.cc:2454] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58ac3de4ab214f0e96ad6ec4aa922eb5 in term 0.
I20260807 11:19:59.598892 18518 raft_consensus.cc:2454] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 1.
I20260807 11:19:59.598878 18379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" is_pre_election: true
I20260807 11:19:59.599195 18379 raft_consensus.cc:2454] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 0.
I20260807 11:19:59.599524 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3bd517cc9dba482384feb361d28126fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
I20260807 11:19:59.599735 18380 raft_consensus.cc:3037] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:19:59.599941 18588 leader_election.cc:302] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd517cc9dba482384feb361d28126fe, 58ac3de4ab214f0e96ad6ec4aa922eb5; no voters:
I20260807 11:19:59.600294 18735 raft_consensus.cc:2781] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:19:59.600602 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.570375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:39365 (local address 127.2.223.131:39611)
0807 11:19:59.571081 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.571087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.585923 (+ 14836us) server_negotiation.cc:407] Connection header received
0807 11:19:59.586198 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.586206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.586304 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.586399 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.592352 (+ 5953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.593515 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.596609 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.598958 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.599087 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.599179 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.600198 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.600333 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":550}
I20260807 11:19:59.600946 18449 leader_election.cc:302] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d9ddca38ca84f1b9f765297cf63cf9b, 58ac3de4ab214f0e96ad6ec4aa922eb5; no voters:
I20260807 11:19:59.601225 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.582414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:52733 (local address 127.2.223.131:39611)
0807 11:19:59.591253 (+ 8839us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.591260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.592087 (+ 827us) server_negotiation.cc:407] Connection header received
0807 11:19:59.592149 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.592154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.592220 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.592573 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:19:59.593400 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.594805 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.597206 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.599335 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.599522 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.599600 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.600461 (+ 861us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.600640 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":688,"server-negotiator.queue_time_us":8672,"thread_start_us":8525,"threads_started":1}
I20260807 11:19:59.601336 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.581944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:39611 (local address 127.2.223.129:52733)
0807 11:19:59.591851 (+ 9907us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.591874 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.591988 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.592893 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.592897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.592924 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.593201 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.593211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.594962 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.594967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.597025 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.597042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.599744 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.599754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.599978 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.600015 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.600171 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.600315 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9658,"thread_start_us":901,"threads_started":1}
I20260807 11:19:59.601444 18737 raft_consensus.cc:2741] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 11:19:59.602406 18663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd517cc9dba482384feb361d28126fe" is_pre_election: true
I20260807 11:19:59.602995 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.570565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:39611 (local address 127.2.223.130:39365)
0807 11:19:59.585833 (+ 15268us) negotiation.cc:107] Waiting for socket to connect
0807 11:19:59.585855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:19:59.586059 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:19:59.591740 (+ 5681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:19:59.591746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:19:59.591825 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:19:59.592166 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.592178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.593736 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.593741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.596265 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:19:59.596280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.599815 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.599821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:19:59.599988 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.600019 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 11:19:59.600769 (+ 750us) client_negotiation.cc:240] Negotiation successful
0807 11:19:59.600921 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15100,"spinlock_wait_cycles":29440,"thread_start_us":7548,"threads_started":1}
I20260807 11:19:59.603204 18380 raft_consensus.cc:2454] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd517cc9dba482384feb361d28126fe in term 1.
I20260807 11:19:59.603549 18735 raft_consensus.cc:686] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 1 LEADER]: Becoming Leader. State: Replica: 3bd517cc9dba482384feb361d28126fe, State: Running, Role: LEADER
I20260807 11:19:59.603820 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a20535a60e44eb8ce71e6769115c1a" candidate_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd517cc9dba482384feb361d28126fe" is_pre_election: true
I20260807 11:19:59.604233 18735 consensus_queue.cc:237] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } }
I20260807 11:19:59.608108 18310 leader_election.cc:302] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d9ddca38ca84f1b9f765297cf63cf9b; no voters: 3bd517cc9dba482384feb361d28126fe, 58ac3de4ab214f0e96ad6ec4aa922eb5
I20260807 11:19:59.608566 18738 raft_consensus.cc:2726] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:19:59.609277 18229 catalog_manager.cc:5495] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd517cc9dba482384feb361d28126fe (127.2.223.131). New cstate: current_term: 1 leader_uuid: "3bd517cc9dba482384feb361d28126fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd517cc9dba482384feb361d28126fe" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 } health_report { overall_health: UNKNOWN } } }
I20260807 11:19:59.685765 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.678956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.2.223.131:39611)
0807 11:19:59.679167 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.679172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.679193 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:19:59.679313 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.679319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.679434 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.679583 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:19:59.680010 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.681087 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.683089 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.683742 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.683867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.684541 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:19:59.685160 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:19:59.685296 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.685400 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.685490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:19:59.694016 18518 raft_consensus.cc:1261] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Refusing update from remote peer 3bd517cc9dba482384feb361d28126fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:59.694265 18380 raft_consensus.cc:1261] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Refusing update from remote peer 3bd517cc9dba482384feb361d28126fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:19:59.694962 18746 consensus_queue.cc:1035] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 40865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:59.695351 18735 consensus_queue.cc:1035] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9ddca38ca84f1b9f765297cf63cf9b" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:19:59.706799 18748 mvcc.cc:205] Tried to move back new op lower bound from 7315872152337268736 to 7315872151983595520. Current Snapshot: MvccSnapshot[applied={T|T < 7315872152337268736}]
I20260807 11:19:59.722992 18226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:59.723280 18226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:19:59.727115 18226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Sending DeleteTablet for 3 replicas of tablet 53a20535a60e44eb8ce71e6769115c1a
I20260807 11:19:59.728266 18642 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a20535a60e44eb8ce71e6769115c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:59 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:59896
I20260807 11:19:59.728468 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 18268
I20260807 11:19:59.728487 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a20535a60e44eb8ce71e6769115c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:59 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:40036
I20260807 11:19:59.729741 18760 tablet_replica.cc:330] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: stopping tablet replica
I20260807 11:19:59.729683 18498 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a20535a60e44eb8ce71e6769115c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:19:59 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:55824
I20260807 11:19:59.730326 18760 raft_consensus.cc:2229] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:59.730393 18759 tablet_replica.cc:330] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: stopping tablet replica
I20260807 11:19:59.730914 18760 raft_consensus.cc:2258] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:59.730891 18759 raft_consensus.cc:2229] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:19:59.732116 18759 raft_consensus.cc:2258] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:59.733299 18760 ts_tablet_manager.cc:1905] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:59.733690 18762 tablet_replica.cc:330] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: stopping tablet replica
I20260807 11:19:59.734771 18762 raft_consensus.cc:2229] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:19:59.735325 18762 raft_consensus.cc:2258] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:19:59.735704 18759 ts_tablet_manager.cc:1905] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:59.740473 18762 ts_tablet_manager.cc:1905] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:19:59.741427 18760 ts_tablet_manager.cc:1918] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:59.741705 18760 log.cc:1196] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/53a20535a60e44eb8ce71e6769115c1a
I20260807 11:19:59.742528 18760 ts_tablet_manager.cc:1939] T 53a20535a60e44eb8ce71e6769115c1a P 3d9ddca38ca84f1b9f765297cf63cf9b: Deleting consensus metadata
I20260807 11:19:59.743372 18759 ts_tablet_manager.cc:1918] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:59.743547 18759 log.cc:1196] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/53a20535a60e44eb8ce71e6769115c1a
I20260807 11:19:59.744127 18759 ts_tablet_manager.cc:1939] T 53a20535a60e44eb8ce71e6769115c1a P 3bd517cc9dba482384feb361d28126fe: Deleting consensus metadata
I20260807 11:19:59.744722 18211 catalog_manager.cc:4841] TS 3d9ddca38ca84f1b9f765297cf63cf9b (127.2.223.129:34803): tablet 53a20535a60e44eb8ce71e6769115c1a (table test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]) successfully deleted
I20260807 11:19:59.745865 18212 catalog_manager.cc:4841] TS 3bd517cc9dba482384feb361d28126fe (127.2.223.131:39611): tablet 53a20535a60e44eb8ce71e6769115c1a (table test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]) successfully deleted
I20260807 11:19:59.748689 18762 ts_tablet_manager.cc:1918] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:19:59.748868 18762 log.cc:1196] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/53a20535a60e44eb8ce71e6769115c1a
I20260807 11:19:59.749529 18762 ts_tablet_manager.cc:1939] T 53a20535a60e44eb8ce71e6769115c1a P 58ac3de4ab214f0e96ad6ec4aa922eb5: Deleting consensus metadata
I20260807 11:19:59.751195 18211 catalog_manager.cc:4841] TS 58ac3de4ab214f0e96ad6ec4aa922eb5 (127.2.223.130:40865): tablet 53a20535a60e44eb8ce71e6769115c1a (table test-table [id=e9411cfabc7c4f0dbd3782d507464dcd]) successfully deleted
I20260807 11:19:59.758265 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.729426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.2.223.129:34803)
0807 11:19:59.729710 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.729719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.734718 (+ 4999us) server_negotiation.cc:407] Connection header received
0807 11:19:59.735900 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.735906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.736019 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.736118 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:59.741429 (+ 5311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.742758 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.747716 (+ 4958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.748434 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.748619 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.752303 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:59.752315 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:59.752329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:59.752361 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:59.755319 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.756146 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.756153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.756155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.756297 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.756704 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.756707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.756709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.757034 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:59.757255 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.757675 (+ 420us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.757876 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:19:59.758924 18424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:59.855274 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 18268
I20260807 11:19:59.879204 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 18433
I20260807 11:19:59.895573 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:19:59.880054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.2.223.130:40865)
0807 11:19:59.880240 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:19:59.880247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:19:59.880270 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:19:59.880460 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:19:59.880467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:19:59.880581 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:19:59.880697 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:19:59.882060 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.883351 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.884893 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:19:59.885531 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:19:59.885666 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:19:59.888381 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:19:59.888399 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:19:59.888420 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:19:59.888465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:19:59.892455 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.893167 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.893177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.893181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.893347 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:19:59.893782 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:19:59.893787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:19:59.893791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:19:59.894247 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:19:59.894608 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 11:19:59.894951 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 11:19:59.895182 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:19:59.896390 18562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:19:59.991969 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 18433
I20260807 11:20:00.013681 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 18571
I20260807 11:20:00.030366 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:00.014680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.2.223.131:39611)
0807 11:20:00.014913 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:00.014919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:00.014969 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:20:00.015166 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:00.015173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:00.015282 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:00.015412 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:00.016778 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:00.018018 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:00.019485 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:00.020170 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:00.020310 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:00.022969 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:00.022986 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:00.023006 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:00.023046 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:00.027289 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:00.028098 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:00.028108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:00.028112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:00.028307 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:00.028692 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:00.028697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:00.028701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:00.029144 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:00.029502 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:00.029740 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 11:20:00.029968 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:20:00.031213 18704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:00.156587 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 18571
I20260807 11:20:00.181686 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 18193
I20260807 11:20:00.182899 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:00.299530 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 18193
I20260807 11:20:00.326263 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--webserver_interface=127.2.223.190
--webserver_port=43495
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:00.667461 18767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:00.667800 18767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:00.667968 18767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:00.674981 18767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:20:00.675134 18767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:00.675241 18767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:00.675343 18767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:20:00.675426 18767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:20:00.684923 18767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:00.686299 18767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:00.687598 18767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:00.699338 18772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:00.699437 18773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:00.699462 18775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:00.700547 18767 server_base.cc:1034] running on GCE node
I20260807 11:20:00.701283 18767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:00.702636 18767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:00.703864 18767 hybrid_clock.cc:648] HybridClock initialized: now 1786101600703818 us; error 58 us; skew 500 ppm
I20260807 11:20:00.704216 18767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:00 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:00.707304 18767 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:20:00.708781 18767 webserver.cc:460] Webserver started at http://127.2.223.190:43495/ using document root <none> and password file <none>
I20260807 11:20:00.709440 18767 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:00.709597 18767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:00.715058 18767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:20:00.719823 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:00.721129 18767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:20:00.721311 18767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "e7d58cb3825c4dd388e08a583385853b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "6387abecb9530524fa223e38ecfb9b73"
server_key_iv: "79a56fa2d54ddc8bee66bf67c9e1f8d1"
server_key_version: "encryptionkey@0"
I20260807 11:20:00.722266 18767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:00.757088 18767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:00.761473 18767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:00.761771 18767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:00.780580 18767 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:39045
I20260807 11:20:00.780602 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:39045 every 8 connection(s)
I20260807 11:20:00.782058 18767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:20:00.786458 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18767
I20260807 11:20:00.787516 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:34803
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=38629
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:20:00.790011 18839 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:20:00.795279 18839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap starting.
I20260807 11:20:00.818027 18839 log.cc:824] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:00.834944 18839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:20:00.835486 18839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b: Bootstrap complete.
I20260807 11:20:00.844751 18839 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:20:00.845623 18839 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Initialized, Role: FOLLOWER
I20260807 11:20:00.846238 18839 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:20:00.846482 18839 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:20:00.846645 18839 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:20:00.846849 18839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:20:00.853237 18839 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:20:00.853937 18839 leader_election.cc:302] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d58cb3825c4dd388e08a583385853b; no voters:
I20260807 11:20:00.854785 18839 leader_election.cc:288] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 11:20:00.855019 18843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:20:00.858378 18843 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [term 3 LEADER]: Becoming Leader. State: Replica: e7d58cb3825c4dd388e08a583385853b, State: Running, Role: LEADER
I20260807 11:20:00.859097 18843 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } }
I20260807 11:20:00.860278 18839 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:20:00.863644 18844 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d58cb3825c4dd388e08a583385853b. Latest consensus state: current_term: 3 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:20:00.863986 18844 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:00.867187 18845 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7d58cb3825c4dd388e08a583385853b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d58cb3825c4dd388e08a583385853b" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39045 } } }
I20260807 11:20:00.867434 18845 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d58cb3825c4dd388e08a583385853b [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:00.867986 18850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:20:00.880447 18850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:20:00.881958 18850 catalog_manager.cc:1259] Loaded cluster ID: 65df56d7ffc0460f93828cbe91333d26
I20260807 11:20:00.882066 18850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 11:20:00.884163 18850 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd28026c9ad google::LogMessage::Fail() at ??:0
@ 0x7fd28026f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd28026c4de google::LogMessage::Flush() at ??:0
@ 0x7fd28026da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd295c9f512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd295c9c976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd295d24610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd295d24439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd282eb8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd282ebb91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd282ebb799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd282e96eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2861706db start_thread at ??:0
@ 0x7fd27f10661f clone at ??:0
W20260807 11:20:01.237246 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:01.237555 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:01.237715 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:01.245788 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:01.245944 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:01.246121 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:20:01.256242 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=38629
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:01.257588 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:01.258997 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:01.272439 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:01.272473 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:01.272365 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:01.273298 18841 server_base.cc:1034] running on GCE node
I20260807 11:20:01.274084 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:01.275969 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:01.277217 18841 hybrid_clock.cc:648] HybridClock initialized: now 1786101601277156 us; error 51 us; skew 500 ppm
I20260807 11:20:01.277645 18841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:01 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:01.281308 18841 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:20:01.282938 18841 webserver.cc:460] Webserver started at http://127.2.223.129:38629/ using document root <none> and password file <none>
I20260807 11:20:01.283710 18841 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:01.283860 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:01.289659 18841 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 11:20:01.292527 18894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.293836 18841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:20:01.294009 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "3d9ddca38ca84f1b9f765297cf63cf9b"
format_stamp: "Formatted at 2026-08-07 11:19:54 on dist-test-slave-5l3k"
server_key: "c5680442316ae3c88a390db730f3a68e"
server_key_iv: "aca7b3ffb55fcda2a43f48b21ab6092f"
server_key_version: "encryptionkey@0"
I20260807 11:20:01.295030 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:01.312083 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:01.316385 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:01.316601 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:01.317541 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:01.318841 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:01.318940 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.319013 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:01.319078 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.352649 18841 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:34803
I20260807 11:20:01.352752 19016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:34803 every 8 connection(s)
I20260807 11:20:01.354138 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:20:01.355104 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 18841
I20260807 11:20:01.355947 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:40865
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=40451
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:20:01.359828 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.357486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:43979)
0807 11:20:01.358373 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.358688 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
W20260807 11:20:01.360571 19017 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:39045 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:39045: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
I20260807 11:20:01.361559 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.361103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:55801)
0807 11:20:01.361296 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.361389 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 11:20:01.362555 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.362094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:43079)
0807 11:20:01.362269 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.362371 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:20:01.362803 19017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:20:01.700903 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:01.701241 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:01.701418 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:01.708710 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:01.708873 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:01.709061 19020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:20:01.719067 19020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:40865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=40451
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:01.720350 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:01.721728 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:01.735162 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:01.735196 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:01.735186 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:01.735986 19020 server_base.cc:1034] running on GCE node
I20260807 11:20:01.736696 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:01.737599 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:01.738831 19020 hybrid_clock.cc:648] HybridClock initialized: now 1786101601738797 us; error 54 us; skew 500 ppm
I20260807 11:20:01.739140 19020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:01 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:01.742223 19020 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:20:01.743714 19020 webserver.cc:460] Webserver started at http://127.2.223.130:40451/ using document root <none> and password file <none>
I20260807 11:20:01.744362 19020 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:01.744464 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:01.749596 19020 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:20:01.752318 19036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.753638 19020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:20:01.753818 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "58ac3de4ab214f0e96ad6ec4aa922eb5"
format_stamp: "Formatted at 2026-08-07 11:19:55 on dist-test-slave-5l3k"
server_key: "3da562781686adc2d55809f113e48da8"
server_key_iv: "595cb4c11d14d9ca8074c14344aef86b"
server_key_version: "encryptionkey@0"
I20260807 11:20:01.754923 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:01.774385 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:01.778687 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:01.778973 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:01.780010 19020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:01.781376 19020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:01.781502 19020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.781646 19020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:01.781749 19020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:01.816848 19020 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:40865
I20260807 11:20:01.816921 19158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:40865 every 8 connection(s)
I20260807 11:20:01.818405 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:20:01.823315 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.821347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:56975)
0807 11:20:01.822079 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.822329 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":291,"threads_started":1}
W20260807 11:20:01.823988 19159 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:39045 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:39045: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
I20260807 11:20:01.824841 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.824502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:45555)
0807 11:20:01.824664 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.824737 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 11:20:01.825656 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:01.825331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:56139)
0807 11:20:01.825470 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:01.825550 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 11:20:01.825901 19159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:20:01.828362 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19020
I20260807 11:20:01.829023 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:39611
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:39045
--webserver_port=44483
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:02.168056 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:02.168396 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:02.168571 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:02.175738 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:02.175891 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:02.176079 19163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:20:02.185628 19163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:39611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=44483
--tserver_master_addrs=127.2.223.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:02.187019 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:02.188372 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:02.201890 19168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:02.201866 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:02.201813 19169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:02.203140 19163 server_base.cc:1034] running on GCE node
I20260807 11:20:02.203707 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:02.204567 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:02.205765 19163 hybrid_clock.cc:648] HybridClock initialized: now 1786101602205725 us; error 31 us; skew 500 ppm
I20260807 11:20:02.206086 19163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:02 dist-test-slave-5l3k krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:02.209293 19163 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:20:02.210700 19163 webserver.cc:460] Webserver started at http://127.2.223.131:44483/ using document root <none> and password file <none>
I20260807 11:20:02.211382 19163 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:02.211532 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:02.216643 19163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:20:02.219604 19178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:02.220961 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:20:02.221163 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "3bd517cc9dba482384feb361d28126fe"
format_stamp: "Formatted at 2026-08-07 11:19:56 on dist-test-slave-5l3k"
server_key: "50aa1c3ab5bf9ec51450fcd7326ad2a4"
server_key_iv: "d1ed53d8c39c3db15aa2ae50e08f90ff"
server_key_version: "encryptionkey@0"
I20260807 11:20:02.222160 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:02.245725 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:02.250353 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:02.250679 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:02.251721 19163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:02.253093 19163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:02.253190 19163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:02.253261 19163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:02.253317 19163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:02.287748 19163 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:39611
I20260807 11:20:02.287851 19296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:39611 every 8 connection(s)
I20260807 11:20:02.289209 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:20:02.290347 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19163
I20260807 11:20:02.294612 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:02.292691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:47143)
0807 11:20:02.293429 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:02.293699 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":285,"threads_started":1}
W20260807 11:20:02.295305 19297 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:39045 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:39045: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
I20260807 11:20:02.296132 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:02.295803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:56535)
0807 11:20:02.295965 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:02.296035 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 11:20:02.296945 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:02.296623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:50519)
0807 11:20:02.296757 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:02.296836 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 11:20:02.297216 19297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:20:02.364135 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:02.363349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:39073)
0807 11:20:02.363799 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:02.363907 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":215,"threads_started":1}
I20260807 11:20:02.827450 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:02.826435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:58739)
0807 11:20:02.827105 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:02.827218 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":285,"threads_started":1}
I20260807 11:20:03.298731 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:03.297806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:47717)
0807 11:20:03.298301 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:03.298393 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260807 11:20:03.365957 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:03.365225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:45411)
0807 11:20:03.365700 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:03.365787 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":237,"threads_started":1}
I20260807 11:20:03.829195 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:03.828463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:59577)
0807 11:20:03.828920 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:03.829002 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":224,"threads_started":1}
I20260807 11:20:04.300746 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:04.299987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:47789)
0807 11:20:04.300479 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:04.300571 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":253,"threads_started":1}
I20260807 11:20:04.367708 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:04.366944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:40779)
0807 11:20:04.367420 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:04.367510 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260807 11:20:04.831066 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:04.830247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:42289)
0807 11:20:04.830777 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:04.830857 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":323,"threads_started":1}
I20260807 11:20:05.302294 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:05.301552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:40263)
0807 11:20:05.302032 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:05.302112 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
I20260807 11:20:05.369519 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:05.368692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:44919)
0807 11:20:05.369183 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:05.369284 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":233,"threads_started":1}
I20260807 11:20:05.832752 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:05.831971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:40441)
0807 11:20:05.832465 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:05.832544 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":244,"threads_started":1}
I20260807 11:20:06.303908 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:06.303167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:57835)
0807 11:20:06.303621 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:06.303714 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260807 11:20:06.371762 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:06.370584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:45139)
0807 11:20:06.371421 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:06.371560 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":607,"threads_started":1}
I20260807 11:20:06.834596 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:06.833752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:58863)
0807 11:20:06.834272 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:06.834355 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":252,"threads_started":1}
I20260807 11:20:07.305651 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:07.304888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:52349)
0807 11:20:07.305374 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:07.305454 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260807 11:20:07.373379 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:07.372630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:41577)
0807 11:20:07.373118 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:07.373211 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":232,"threads_started":1}
I20260807 11:20:07.836310 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:07.835482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:56981)
0807 11:20:07.835996 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:07.836120 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260807 11:20:08.307497 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:08.306691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:56071)
0807 11:20:08.307158 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:08.307249 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
I20260807 11:20:08.375331 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:08.374457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:37447)
0807 11:20:08.375012 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:08.375093 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":229,"threads_started":1}
I20260807 11:20:08.838124 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:08.837182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:41147)
0807 11:20:08.837800 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:08.837881 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":317,"threads_started":1}
I20260807 11:20:09.309336 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:09.308484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:49539)
0807 11:20:09.309019 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:09.309108 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":226,"threads_started":1}
I20260807 11:20:09.377002 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:09.376278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:53287)
0807 11:20:09.376745 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:09.376829 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260807 11:20:09.840016 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:09.839233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:37867)
0807 11:20:09.839767 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:09.839848 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
I20260807 11:20:10.311035 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:10.310254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:55739)
0807 11:20:10.310794 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:10.310876 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
I20260807 11:20:10.378818 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:10.377987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:46163)
0807 11:20:10.378529 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:10.378629 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":242,"threads_started":1}
I20260807 11:20:10.841797 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:10.840961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:48783)
0807 11:20:10.841534 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:10.841616 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":294,"threads_started":1}
I20260807 11:20:11.312770 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:11.311950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:52967)
0807 11:20:11.312482 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:11.312586 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":261,"threads_started":1}
I20260807 11:20:11.380579 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:11.379813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:41781)
0807 11:20:11.380307 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:11.380397 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":281,"threads_started":1}
I20260807 11:20:11.844141 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:11.843050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:53769)
0807 11:20:11.843653 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:11.843760 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":341,"threads_started":1}
I20260807 11:20:12.314345 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:12.313686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:52689)
0807 11:20:12.314111 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:12.314189 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260807 11:20:12.382350 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:12.381555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:52397)
0807 11:20:12.382054 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:12.382152 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260807 11:20:12.846073 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:12.845164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:44243)
0807 11:20:12.845676 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:12.845779 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":234,"threads_started":1}
I20260807 11:20:13.315946 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:13.315237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:42391)
0807 11:20:13.315659 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:13.315757 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260807 11:20:13.384086 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:13.383338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:58079)
0807 11:20:13.383829 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:13.383915 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
I20260807 11:20:13.847841 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:13.847126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:55235)
0807 11:20:13.847576 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:13.847675 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260807 11:20:14.317564 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:14.316858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:48863)
0807 11:20:14.317327 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:14.317407 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260807 11:20:14.385860 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:14.385044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:48537)
0807 11:20:14.385541 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:14.385655 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":244,"threads_started":1}
I20260807 11:20:14.850147 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:14.848975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:36255)
0807 11:20:14.849676 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:14.849842 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":337,"threads_started":1}
I20260807 11:20:15.319363 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:15.318567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:48023)
0807 11:20:15.319062 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:15.319169 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
I20260807 11:20:15.387588 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:15.386868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:48105)
0807 11:20:15.387313 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:15.387393 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260807 11:20:15.851850 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:15.851112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:32863)
0807 11:20:15.851583 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:15.851664 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260807 11:20:16.321058 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:16.320222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:59171)
0807 11:20:16.320765 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:16.320869 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260807 11:20:16.389531 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:16.388739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:42971)
0807 11:20:16.389213 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:16.389314 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260807 11:20:16.854085 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:16.853268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.130:57423)
0807 11:20:16.853800 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:16.853892 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":296,"threads_started":1}
I20260807 11:20:17.302709 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 18841
I20260807 11:20:17.323112 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:17.322218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.131:56515)
0807 11:20:17.322770 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:17.322929 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":303,"threads_started":1}
I20260807 11:20:17.326555 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:17.303814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.2.223.129:34803)
0807 11:20:17.304439 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:17.304454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:17.304496 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:20:17.306783 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:17.306793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:17.307254 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:17.307562 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:17.308975 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.310188 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.311525 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.312174 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.312330 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:17.319884 (+ 7554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:17.319892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:17.319908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:17.319950 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:17.323235 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.324003 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.324009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.324016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.324169 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.324533 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.324536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.324538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.325116 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:17.325346 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:17.325744 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 11:20:17.326233 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
I20260807 11:20:17.328042 19011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:17.416574 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:17.415128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:39045 (local address 127.2.223.129:51181)
0807 11:20:17.415973 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:17.416261 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":398,"threads_started":1}
I20260807 11:20:17.418670 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 18841
I20260807 11:20:17.441952 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19020
I20260807 11:20:17.460157 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:17.443046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.2.223.130:40865)
0807 11:20:17.443632 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:17.443646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:17.443685 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:20:17.445251 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:17.445261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:17.445698 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:17.446000 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:17.447446 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.448641 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.449916 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.450678 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.450818 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:17.453612 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:17.453620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:17.453635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:17.453691 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:17.456684 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.457409 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.457416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.457423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.457631 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.458034 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.458038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.458040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.458695 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:17.458963 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:17.459428 (+ 465us) server_negotiation.cc:299] Negotiation successful
0807 11:20:17.459814 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":268,"threads_started":1}
I20260807 11:20:17.461555 19154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:17.545814 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19020
I20260807 11:20:17.569026 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19163
I20260807 11:20:17.587556 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:17.570164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.2.223.131:39611)
0807 11:20:17.570847 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:17.570861 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:17.570906 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:20:17.572678 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:17.572688 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:17.573136 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:17.573443 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:17.575072 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.576316 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.577570 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:17.578228 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:17.578367 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:17.581074 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:17.581082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:17.581097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:17.581148 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:17.584586 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.585261 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.585267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.585274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.585437 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:17.585759 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:17.585762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:17.585764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:17.586308 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:17.586615 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:17.586923 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:20:17.587256 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":334,"threads_started":1}
I20260807 11:20:17.589003 19292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:17.675201 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19163
2026-08-07T11:20:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25176 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:20:17 dist-test-slave-5l3k krb5kdc[19357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:20:17 dist-test-slave-5l3k krb5kdc[19357](info): set up 2 sockets
Aug 07 11:20:17 dist-test-slave-5l3k krb5kdc[19357](info): commencing operation
krb5kdc: starting...
W20260807 11:20:18.844159 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:20:19 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:20:19.162406 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:19.511493 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:19.511803 19371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:19.511953 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:19.519335 19371 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:20:19.519428 19371 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:20:19.519519 19371 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:20:19.519616 19371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:20:19.519712 19371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:19.519837 19371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:19.519915 19371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:20:19.519997 19371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:20:19.529899 19371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:19.531244 19371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:19.532635 19371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:19.545058 19376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:19.545006 19379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:19.545086 19377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:19.545959 19371 server_base.cc:1034] running on GCE node
I20260807 11:20:19.546712 19371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:19.547987 19371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:19.549229 19371 hybrid_clock.cc:648] HybridClock initialized: now 1786101619549166 us; error 30 us; skew 500 ppm
I20260807 11:20:19.549587 19371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:19 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:19.552838 19371 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:20:19.554332 19371 webserver.cc:460] Webserver started at http://127.2.223.190:45773/ using document root <none> and password file <none>
I20260807 11:20:19.555100 19371 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:19.555193 19371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:19.555461 19371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:20:19.558796 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "77f8804e53784608bb90c7bd7101bde0"
format_stamp: "Formatted at 2026-08-07 11:20:19 on dist-test-slave-5l3k"
server_key: "62341a03e5cd7c4530232296d48b64bb"
server_key_iv: "abc9515059a319d0d0ba3fc0b62573fb"
server_key_version: "encryptionkey@0"
I20260807 11:20:19.559558 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "77f8804e53784608bb90c7bd7101bde0"
format_stamp: "Formatted at 2026-08-07 11:20:19 on dist-test-slave-5l3k"
server_key: "62341a03e5cd7c4530232296d48b64bb"
server_key_iv: "abc9515059a319d0d0ba3fc0b62573fb"
server_key_version: "encryptionkey@0"
I20260807 11:20:19.564671 19371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 11:20:19.568058 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:19.569406 19371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:20:19.569593 19371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "77f8804e53784608bb90c7bd7101bde0"
format_stamp: "Formatted at 2026-08-07 11:20:19 on dist-test-slave-5l3k"
server_key: "62341a03e5cd7c4530232296d48b64bb"
server_key_iv: "abc9515059a319d0d0ba3fc0b62573fb"
server_key_version: "encryptionkey@0"
I20260807 11:20:19.569810 19371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:19.591459 19371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:19.595645 19371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:19.595952 19371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:19.615163 19371 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:45069
I20260807 11:20:19.615190 19439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:45069 every 8 connection(s)
I20260807 11:20:19.616554 19371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:20:19.620234 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:19.624001 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19371
I20260807 11:20:19.624240 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:20:19.624956 2942 external_mini_cluster.cc:1428] Setting key 481e3029cfe7566f1a0908bcfea14e91
I20260807 11:20:19.631343 19440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap starting.
I20260807 11:20:19.634924 19440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:19.636112 19440 log.cc:824] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:19.639165 19440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: No bootstrap required, opened a new log
Aug 07 11:20:19 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:19.649027 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:19.626945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.2.223.190:45069)
0807 11:20:19.627775 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:19.627786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:19.627827 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:20:19.629831 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:19.629853 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:19.630209 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:19.630581 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:19.632486 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:19.633794 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:19.635823 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:19.636768 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:19.637010 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:19.641672 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:19.641682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:19.641695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:19.641731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:19.644909 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:19.645812 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:19.645818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:19.645823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:19.646001 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:19.646433 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:19.646436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:19.646438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:19.646985 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:19.647244 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:19.647711 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 11:20:19.648031 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":398,"threads_started":1}
I20260807 11:20:19.648947 19440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:19.649444 19440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:19.649590 19440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Initialized, Role: FOLLOWER
I20260807 11:20:19.650252 19440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:19.650648 19440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:20:19.650810 19440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:20:19.651011 19440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:19.653764 19440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:19.654209 19440 leader_election.cc:302] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f8804e53784608bb90c7bd7101bde0; no voters:
I20260807 11:20:19.654851 19440 leader_election.cc:288] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:20:19.654945 19445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:20:19.655714 19445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 LEADER]: Becoming Leader. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Running, Role: LEADER
I20260807 11:20:19.656283 19445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:19.657094 19440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:20:19.659348 19446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:20:19.659864 19446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:19.659463 19447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f8804e53784608bb90c7bd7101bde0. Latest consensus state: current_term: 1 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:20:19.660418 19447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:19.660969 19455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:20:19.667795 19455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:20:19.677889 19455 catalog_manager.cc:1347] Generated new cluster ID: d3defa5eee34452bb9986f088f88f83f
I20260807 11:20:19.678004 19455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:20:19.695333 19455 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:20:19.697274 19455 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:20:19.711220 19455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Generated new TSK 0
I20260807 11:20:19.712138 19455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:20:19.810708 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:20.166230 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:20.166559 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:20.166675 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:20.173888 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:20.174000 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:20.174126 19469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:20:20.184399 19469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:20.185705 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:20.187139 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:20.200592 19478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:20.200665 19476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:20.200690 19475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:20.201479 19469 server_base.cc:1034] running on GCE node
I20260807 11:20:20.202144 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:20.203212 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:20.204447 19469 hybrid_clock.cc:648] HybridClock initialized: now 1786101620204386 us; error 38 us; skew 500 ppm
I20260807 11:20:20.204826 19469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:20 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:20.208333 19469 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:20:20.210088 19469 webserver.cc:460] Webserver started at http://127.2.223.129:32905/ using document root <none> and password file <none>
I20260807 11:20:20.210865 19469 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:20.210992 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:20.211369 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:20:20.214572 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "c74da024cd1349d99078a6291163f9b3"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "3a3d50444babdca9456c76a0b440f4d8"
server_key_iv: "739c0f32fc17fe62c629c94fc0da3c3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.215353 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "c74da024cd1349d99078a6291163f9b3"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "3a3d50444babdca9456c76a0b440f4d8"
server_key_iv: "739c0f32fc17fe62c629c94fc0da3c3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.221007 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 11:20:20.224601 19485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.226082 19469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:20:20.226286 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "c74da024cd1349d99078a6291163f9b3"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "3a3d50444babdca9456c76a0b440f4d8"
server_key_iv: "739c0f32fc17fe62c629c94fc0da3c3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.226490 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:20.249920 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:20.254107 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:20.254413 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:20.255455 19469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:20.256865 19469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:20.256942 19469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.257123 19469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:20.257185 19469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.291954 19469 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:45739
I20260807 11:20:20.292014 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:45739 every 8 connection(s)
I20260807 11:20:20.293308 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:20:20.298167 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19469
I20260807 11:20:20.298408 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:20:20.299151 2942 external_mini_cluster.cc:1428] Setting key 10177a6e6181f6836f465c8a9e6adef2
Aug 07 11:20:20 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:20.313702 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:20.295941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:36367 (local address 127.2.223.190:45069)
0807 11:20:20.296428 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:20.296433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:20.297303 (+ 870us) server_negotiation.cc:407] Connection header received
0807 11:20:20.299418 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:20.299424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:20.299518 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:20.299640 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:20.301178 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.301897 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.303530 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.303961 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.304105 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:20.307803 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:20.307808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:20.307811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:20.307856 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:20.310616 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:20.311364 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:20.311369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:20.311377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:20.311499 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:20.311941 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:20.311946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:20.311949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:20.312223 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:20.312381 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:20.313101 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 11:20:20.313343 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260807 11:20:20.314734 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:20.296349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:36367)
0807 11:20:20.297129 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:20.297175 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:20.299094 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:20.299897 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:20.299905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:20.300425 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:20.300977 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:20.300995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.302062 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.302066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:20.303380 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:20.303392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.304218 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.304226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:20.304451 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:20.305139 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:20.305162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:20.307581 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:20.310756 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:20.310765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:20.310781 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:20.311230 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:20.311612 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:20.311617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:20.311620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:20.311816 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:20.312362 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:20.312369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:20.312707 (+ 338us) client_negotiation.cc:769] Sending connection context
0807 11:20:20.312979 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 11:20:20.313505 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":297,"threads_started":1}
I20260807 11:20:20.316398 19608 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:20.316744 19608 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:20.317602 19608 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:20.320202 19404 ts_manager.cc:194] Registered new tserver with Master: c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739)
I20260807 11:20:20.322444 19404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:36367
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:20:20.370466 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:20.717207 19614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:20.717476 19614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:20.717572 19614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:20.724728 19614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:20.724829 19614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:20.724936 19614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:20:20.734536 19614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:20.735689 19614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:20.736907 19614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:20.750162 19623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:20.750272 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:20.750247 19621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:20.750849 19614 server_base.cc:1034] running on GCE node
I20260807 11:20:20.751412 19614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:20.752228 19614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:20.753423 19614 hybrid_clock.cc:648] HybridClock initialized: now 1786101620753388 us; error 34 us; skew 500 ppm
I20260807 11:20:20.753686 19614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:20 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:20.756824 19614 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:20:20.758337 19614 webserver.cc:460] Webserver started at http://127.2.223.130:44819/ using document root <none> and password file <none>
I20260807 11:20:20.759056 19614 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:20.759191 19614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:20.759567 19614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:20:20.762428 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "d313f5cbb893410a97819e969c03974b"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "e8fb548b269022ba45927168715256cf"
server_key_iv: "6aa61f5c2043c1ba2a70e6a009abf2d2"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.763322 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "d313f5cbb893410a97819e969c03974b"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "e8fb548b269022ba45927168715256cf"
server_key_iv: "6aa61f5c2043c1ba2a70e6a009abf2d2"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.768355 19614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:20:20.771835 19630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.773144 19614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:20:20.773336 19614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "d313f5cbb893410a97819e969c03974b"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "e8fb548b269022ba45927168715256cf"
server_key_iv: "6aa61f5c2043c1ba2a70e6a009abf2d2"
server_key_version: "encryptionkey@0"
I20260807 11:20:20.773545 19614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:20.785880 19614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:20.789633 19614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:20.789909 19614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:20.790983 19614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:20.792202 19614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:20.792306 19614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.792454 19614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:20.792538 19614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:20.826858 19614 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42091
I20260807 11:20:20.826906 19752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42091 every 8 connection(s)
I20260807 11:20:20.828176 19614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:20:20.831817 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19614
I20260807 11:20:20.832055 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:20:20.832681 2942 external_mini_cluster.cc:1428] Setting key c2d17ea10cba08906fb85b425b787ce5
Aug 07 11:20:20 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:20.852324 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:20.830945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:38659 (local address 127.2.223.190:45069)
0807 11:20:20.831610 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:20.831616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:20.832579 (+ 963us) server_negotiation.cc:407] Connection header received
0807 11:20:20.835017 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:20.835023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:20.835106 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:20.835227 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:20.836825 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.838013 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.839614 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.840349 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.840493 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:20.844235 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:20.844244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:20.844251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:20.844297 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:20.847957 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:20.849143 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:20.849152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:20.849156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:20.849374 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:20.849859 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:20.849864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:20.849868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:20.850138 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:20.850297 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:20.851386 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0807 11:20:20.851734 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":353,"threads_started":1}
I20260807 11:20:20.853255 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:20.831576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:38659)
0807 11:20:20.832418 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:20.832457 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:20.834706 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:20.835446 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:20.835455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:20.835961 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:20.836598 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:20.836626 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.838159 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.838171 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:20.839463 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:20.839476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:20.840568 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:20.840573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:20.840770 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:20.841478 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:20.841501 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:20.843995 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:20.848141 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:20.848152 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:20.848190 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:20.848949 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:20.849508 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:20.849513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:20.849517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:20.849735 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:20.850267 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:20.850275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:20.850818 (+ 543us) client_negotiation.cc:769] Sending connection context
0807 11:20:20.851208 (+ 390us) client_negotiation.cc:240] Negotiation successful
0807 11:20:20.851760 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":364,"threads_started":1}
I20260807 11:20:20.855293 19753 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:20.855665 19753 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:20.856623 19753 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:20.858425 19404 ts_manager.cc:194] Registered new tserver with Master: d313f5cbb893410a97819e969c03974b (127.2.223.130:42091)
I20260807 11:20:20.859282 19404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:38659
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:20:20.909813 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:21.248080 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:21.248345 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:21.248440 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:21.255579 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:21.255690 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:21.255800 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:20:21.265894 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:21.267148 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:21.268402 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:21.282516 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:21.282594 19769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:21.282557 19766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:21.287242 19761 server_base.cc:1034] running on GCE node
I20260807 11:20:21.288138 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:21.289314 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:21.290576 19761 hybrid_clock.cc:648] HybridClock initialized: now 1786101621290511 us; error 52 us; skew 500 ppm
I20260807 11:20:21.290997 19761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:21 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:21.298122 19761 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:20:21.300096 19761 webserver.cc:460] Webserver started at http://127.2.223.131:36081/ using document root <none> and password file <none>
I20260807 11:20:21.301105 19761 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:21.301252 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:21.301651 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:20:21.306077 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
format_stamp: "Formatted at 2026-08-07 11:20:21 on dist-test-slave-5l3k"
server_key: "52e59692f9002263e890951cfe3d7797"
server_key_iv: "2bf8462ae3a73c97a28d46789228ad3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:21.307092 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
format_stamp: "Formatted at 2026-08-07 11:20:21 on dist-test-slave-5l3k"
server_key: "52e59692f9002263e890951cfe3d7797"
server_key_iv: "2bf8462ae3a73c97a28d46789228ad3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:21.314733 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 11:20:21.319618 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:21.321342 19761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:20:21.321512 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
format_stamp: "Formatted at 2026-08-07 11:20:21 on dist-test-slave-5l3k"
server_key: "52e59692f9002263e890951cfe3d7797"
server_key_iv: "2bf8462ae3a73c97a28d46789228ad3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:21.321736 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:21.326288 19608 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:21.338112 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:21.342643 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:21.342935 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:21.343927 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:21.345297 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:21.345377 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:21.345542 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:21.345598 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:21.380833 19761 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:33749
I20260807 11:20:21.380867 19895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:33749 every 8 connection(s)
I20260807 11:20:21.382205 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:20:21.391157 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19761
I20260807 11:20:21.391371 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:20:21.391851 2942 external_mini_cluster.cc:1428] Setting key 78cfbcb8d32a0849c2babf36d4175dbd
Aug 07 11:20:21 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:21.402042 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.384656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:39917 (local address 127.2.223.190:45069)
0807 11:20:21.385170 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.385178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.386121 (+ 943us) server_negotiation.cc:407] Connection header received
0807 11:20:21.388018 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.388023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.388092 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.388196 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:21.389629 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.390436 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.392180 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.392705 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.392837 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.396737 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:21.396742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:21.396745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:21.396782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:21.399405 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.400089 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.400093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.400096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.400217 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.400600 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.400604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.400606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.400863 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:21.400982 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.401602 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.401759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260807 11:20:21.402977 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.385214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:39917)
0807 11:20:21.385949 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.385995 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.387741 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.388405 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.388413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.388919 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:21.389466 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.389484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.390635 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.390639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.392037 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.392050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.392931 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.392939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.393217 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.393933 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:21.393965 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:21.396538 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:21.399532 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:21.399538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:21.399549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:21.399961 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:21.400313 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:21.400316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:21.400318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:21.400495 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:21.400971 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:21.400976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:21.401262 (+ 286us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.401479 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.401842 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":278,"threads_started":1}
I20260807 11:20:21.404295 19896 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:21.404584 19896 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:21.405298 19896 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:21.406829 19404 ts_manager.cc:194] Registered new tserver with Master: c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749)
I20260807 11:20:21.407688 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:20:21.407734 19404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:39917
I20260807 11:20:21.424489 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.412455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.2.223.190:45069)
0807 11:20:21.412629 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.412634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.413120 (+ 486us) server_negotiation.cc:407] Connection header received
0807 11:20:21.413271 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.413274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.413342 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.413420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:21.414942 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.415634 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.416972 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.417358 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.417469 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.419944 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:21.419948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:21.419952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:21.419981 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:21.422284 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.422917 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.422921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.422923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.423040 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.423366 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.423369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.423371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.423658 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:21.423842 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.424125 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.424275 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:20:21.428285 19404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:20:21.435328 19404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:20:21.463559 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.448933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.2.223.129:45739)
0807 11:20:21.455139 (+ 6206us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.455147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.455184 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:20:21.455279 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.455285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.455456 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.456107 (+ 651us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.456666 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.458226 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.460495 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462067 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.462216 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.462466 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.463059 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.463241 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6034,"thread_start_us":5912,"threads_started":1}
I20260807 11:20:21.463523 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.447591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:49130)
0807 11:20:21.453594 (+ 6003us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.453612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.453724 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.455672 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.455681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.455716 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.456360 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.456373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.458355 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.458362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.460331 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.460343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462291 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.462298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.462551 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.462628 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.462830 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.463039 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5856,"mutex_wait_us":130,"thread_start_us":6867,"threads_started":1}
I20260807 11:20:21.463825 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.453657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.2.223.131:33749)
0807 11:20:21.454402 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.454413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.454828 (+ 415us) server_negotiation.cc:407] Connection header received
0807 11:20:21.455173 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.455181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.455408 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.455618 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.456783 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.458277 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.461040 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462482 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.462639 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.462944 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.463312 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.463472 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":460,"threads_started":1}
I20260807 11:20:21.463878 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.447371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:42091 (local address 127.0.0.1:33128)
0807 11:20:21.452586 (+ 5215us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.452630 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.452805 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.455502 (+ 2697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.455510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.455540 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.455979 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.455990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.458061 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.458067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.460799 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.460815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462664 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.462673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.462970 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.463112 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.463198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.463353 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4963,"mutex_wait_us":104,"thread_start_us":4808,"threads_started":1}
I20260807 11:20:21.464785 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.453000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33749 (local address 127.0.0.1:60680)
0807 11:20:21.454716 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.454733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.454928 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.455924 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.455934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.455963 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.456539 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.456555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.458466 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.458472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.460867 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.460880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462705 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.462710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.462970 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.463108 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.463202 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.464412 (+ 1210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1592,"mutex_wait_us":1204,"thread_start_us":960,"threads_started":1}
I20260807 11:20:21.464998 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.449439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.2.223.130:42091)
0807 11:20:21.454910 (+ 5471us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.454919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.454945 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:20:21.455048 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.455059 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.455294 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.455498 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.456241 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.457914 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.460970 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.462372 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.464220 (+ 1848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.464436 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.464507 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.464690 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5300,"thread_start_us":5205,"threads_started":1}
I20260807 11:20:21.468150 19683 tablet_service.cc:1467] Processing CreateTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 (DEFAULT_TABLE table=test-table [id=6e04c403cdb54de094c5cb772aff8b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:21.468258 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 (DEFAULT_TABLE table=test-table [id=6e04c403cdb54de094c5cb772aff8b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:21.468163 19537 tablet_service.cc:1467] Processing CreateTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 (DEFAULT_TABLE table=test-table [id=6e04c403cdb54de094c5cb772aff8b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:21.469429 19683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3892ff3a3131478384eeb5c1d435bd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:21.469601 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3892ff3a3131478384eeb5c1d435bd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:21.469619 19537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3892ff3a3131478384eeb5c1d435bd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:21.482352 19916 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Bootstrap starting.
I20260807 11:20:21.484158 19917 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Bootstrap starting.
I20260807 11:20:21.486692 19916 tablet_bootstrap.cc:654] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:21.487725 19917 tablet_bootstrap.cc:654] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:21.488307 19916 log.cc:824] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:21.489008 19917 log.cc:824] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:21.490427 19918 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Bootstrap starting.
I20260807 11:20:21.492537 19917 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: No bootstrap required, opened a new log
I20260807 11:20:21.492866 19917 ts_tablet_manager.cc:1397] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 11:20:21.494832 19918 tablet_bootstrap.cc:654] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:21.495999 19918 log.cc:824] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:21.497238 19916 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: No bootstrap required, opened a new log
I20260807 11:20:21.497539 19916 ts_tablet_manager.cc:1397] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260807 11:20:21.499210 19918 tablet_bootstrap.cc:492] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: No bootstrap required, opened a new log
I20260807 11:20:21.499460 19918 ts_tablet_manager.cc:1397] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260807 11:20:21.501408 19917 raft_consensus.cc:348] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.501787 19917 raft_consensus.cc:374] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:21.501924 19917 raft_consensus.cc:729] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74da024cd1349d99078a6291163f9b3, State: Initialized, Role: FOLLOWER
I20260807 11:20:21.502614 19917 consensus_queue.cc:260] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.504372 19917 ts_tablet_manager.cc:1428] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260807 11:20:21.508440 19916 raft_consensus.cc:348] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.508939 19916 raft_consensus.cc:374] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:21.509044 19916 raft_consensus.cc:729] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d313f5cbb893410a97819e969c03974b, State: Initialized, Role: FOLLOWER
I20260807 11:20:21.509754 19916 consensus_queue.cc:260] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.510296 19918 raft_consensus.cc:348] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.511003 19753 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:21.511703 19916 ts_tablet_manager.cc:1428] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260807 11:20:21.511988 19918 raft_consensus.cc:374] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:21.512122 19918 raft_consensus.cc:729] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c85cbd02d548d6a58d9589a07b4ff3, State: Initialized, Role: FOLLOWER
I20260807 11:20:21.512776 19918 consensus_queue.cc:260] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.514001 19896 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:21.514635 19918 ts_tablet_manager.cc:1428] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260807 11:20:21.545907 19609 tablet.cc:2369] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:20:21.580299 19754 tablet.cc:2369] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:20:21.633898 19897 tablet.cc:2369] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:20:21.720008 19923 raft_consensus.cc:482] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:20:21.720383 19923 raft_consensus.cc:504] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.722391 19923 leader_election.cc:288] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739), c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749)
I20260807 11:20:21.722616 19924 raft_consensus.cc:482] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:20:21.723219 19924 raft_consensus.cc:504] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.727361 19924 leader_election.cc:288] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d313f5cbb893410a97819e969c03974b (127.2.223.130:42091), c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739)
I20260807 11:20:21.733214 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.722904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49949 (local address 127.2.223.129:45739)
0807 11:20:21.723116 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.723123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.723829 (+ 706us) server_negotiation.cc:407] Connection header received
0807 11:20:21.724096 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.724104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.724191 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.724265 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.725158 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.726299 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.730343 (+ 4044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.731647 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.731784 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.731871 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.732769 (+ 898us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.732918 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:20:21.733700 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.723315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:50079 (local address 127.2.223.131:33749)
0807 11:20:21.723548 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.723554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.724464 (+ 910us) server_negotiation.cc:407] Connection header received
0807 11:20:21.724727 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.724732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.724814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.724914 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.725646 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.726833 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.730328 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.731792 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.731935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.732006 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.732908 (+ 902us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.733049 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":77}
I20260807 11:20:21.733747 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.725539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:57999 (local address 127.2.223.130:42091)
0807 11:20:21.726572 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.726578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.726594 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:20:21.726643 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.726647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.726705 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.726790 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.727401 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.728491 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.731055 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.732469 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.732638 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.732713 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.733327 (+ 614us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.733460 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":878}
I20260807 11:20:21.734203 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.725460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:42091 (local address 127.2.223.131:57999)
0807 11:20:21.725770 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.725794 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.725893 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.726825 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.726833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.726873 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.727201 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.727220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.728713 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.728718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.730798 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.730826 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.732637 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.732643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.732931 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.733010 (+ 79us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.733097 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.733233 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":86}
I20260807 11:20:21.734403 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.722865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33749 (local address 127.2.223.130:50079)
0807 11:20:21.724344 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.724358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.724570 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.724937 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.724941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.724966 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.725446 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.725459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.727058 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.727066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.730152 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.730160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.732201 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.732206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.732501 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.732755 (+ 254us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.733879 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.733998 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"mutex_wait_us":927,"spinlock_wait_cycles":48384,"thread_start_us":485,"threads_started":1}
I20260807 11:20:21.734933 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.722865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.2.223.130:49949)
0807 11:20:21.723675 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.723695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.723883 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.724481 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.724486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.724524 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.724935 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.724949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.727743 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.727750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.730142 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.730158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.732126 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.732138 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.732431 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.732607 (+ 176us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.733470 (+ 863us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.733592 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":94,"thread_start_us":414,"threads_started":1}
I20260807 11:20:21.735186 19706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d313f5cbb893410a97819e969c03974b" is_pre_election: true
I20260807 11:20:21.735675 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.726140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.2.223.131:49367)
0807 11:20:21.727424 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:21.727442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:21.727561 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:21.728352 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:21.728357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:21.728383 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:21.728816 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.728828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.730603 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.730609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.733001 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:21.733013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.734828 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.734834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:21.735015 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.735049 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:20:21.735165 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 11:20:21.735292 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":9,"thread_start_us":903,"threads_started":1}
I20260807 11:20:21.735675 19706 raft_consensus.cc:2454] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c85cbd02d548d6a58d9589a07b4ff3 in term 0.
I20260807 11:20:21.735817 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.726203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:49367 (local address 127.2.223.129:45739)
0807 11:20:21.728015 (+ 1812us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.728021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.728043 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:20:21.728095 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.728102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.728202 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.728310 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:21.729031 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.730358 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.733192 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.734546 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.734706 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.734766 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.735308 (+ 542us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.735444 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672,"thread_start_us":1427,"threads_started":1}
I20260807 11:20:21.735914 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "d313f5cbb893410a97819e969c03974b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74da024cd1349d99078a6291163f9b3" is_pre_election: true
I20260807 11:20:21.736516 19559 raft_consensus.cc:2454] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d313f5cbb893410a97819e969c03974b in term 0.
I20260807 11:20:21.736549 19778 leader_election.cc:302] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7c85cbd02d548d6a58d9589a07b4ff3, d313f5cbb893410a97819e969c03974b; no voters:
I20260807 11:20:21.736512 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74da024cd1349d99078a6291163f9b3" is_pre_election: true
I20260807 11:20:21.736908 19558 raft_consensus.cc:2454] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c85cbd02d548d6a58d9589a07b4ff3 in term 0.
I20260807 11:20:21.737025 19924 raft_consensus.cc:2781] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:20:21.737195 19924 raft_consensus.cc:482] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:20:21.736954 19848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "d313f5cbb893410a97819e969c03974b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" is_pre_election: true
I20260807 11:20:21.737293 19924 raft_consensus.cc:3037] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:21.737282 19633 leader_election.cc:302] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c74da024cd1349d99078a6291163f9b3, d313f5cbb893410a97819e969c03974b; no voters:
I20260807 11:20:21.737673 19923 raft_consensus.cc:2781] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:20:21.737845 19923 raft_consensus.cc:482] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:20:21.737958 19923 raft_consensus.cc:3037] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:21.739892 19924 raft_consensus.cc:504] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.740373 19848 raft_consensus.cc:2379] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d313f5cbb893410a97819e969c03974b in current term 1: Already voted for candidate c7c85cbd02d548d6a58d9589a07b4ff3 in this term.
I20260807 11:20:21.740255 19923 raft_consensus.cc:504] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.740622 19924 leader_election.cc:288] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [CANDIDATE]: Term 1 election: Requested vote from peers d313f5cbb893410a97819e969c03974b (127.2.223.130:42091), c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739)
I20260807 11:20:21.741119 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74da024cd1349d99078a6291163f9b3"
I20260807 11:20:21.741302 19706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d313f5cbb893410a97819e969c03974b"
I20260807 11:20:21.741338 19558 raft_consensus.cc:3037] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:21.741562 19706 raft_consensus.cc:2379] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7c85cbd02d548d6a58d9589a07b4ff3 in current term 1: Already voted for candidate d313f5cbb893410a97819e969c03974b in this term.
I20260807 11:20:21.741720 19847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "d313f5cbb893410a97819e969c03974b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
I20260807 11:20:21.741879 19923 leader_election.cc:288] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [CANDIDATE]: Term 1 election: Requested vote from peers c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739), c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749)
I20260807 11:20:21.742064 19847 raft_consensus.cc:2379] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d313f5cbb893410a97819e969c03974b in current term 1: Already voted for candidate c7c85cbd02d548d6a58d9589a07b4ff3 in this term.
I20260807 11:20:21.742828 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3892ff3a3131478384eeb5c1d435bd04" candidate_uuid: "d313f5cbb893410a97819e969c03974b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74da024cd1349d99078a6291163f9b3"
I20260807 11:20:21.743590 19633 leader_election.cc:302] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d313f5cbb893410a97819e969c03974b; no voters: c74da024cd1349d99078a6291163f9b3, c7c85cbd02d548d6a58d9589a07b4ff3
I20260807 11:20:21.743975 19923 raft_consensus.cc:2726] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 11:20:21.744588 19558 raft_consensus.cc:2454] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c85cbd02d548d6a58d9589a07b4ff3 in term 1.
I20260807 11:20:21.745012 19780 leader_election.cc:302] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c74da024cd1349d99078a6291163f9b3, c7c85cbd02d548d6a58d9589a07b4ff3; no voters: d313f5cbb893410a97819e969c03974b
I20260807 11:20:21.745306 19924 raft_consensus.cc:2781] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:20:21.745767 19924 raft_consensus.cc:686] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 LEADER]: Becoming Leader. State: Replica: c7c85cbd02d548d6a58d9589a07b4ff3, State: Running, Role: LEADER
I20260807 11:20:21.746299 19924 consensus_queue.cc:237] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } }
I20260807 11:20:21.750317 19403 catalog_manager.cc:5495] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131). New cstate: current_term: 1 leader_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260807 11:20:21.827540 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.821752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.2.223.131:33749)
0807 11:20:21.821966 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.821971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.821990 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:20:21.822195 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.822199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.822280 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.822406 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:20:21.822825 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.823569 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.825141 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.825558 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.825657 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.826307 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:20:21.826868 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:20:21.826983 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.827173 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.827240 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:20:21.836302 19706 raft_consensus.cc:1261] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Refusing update from remote peer c7c85cbd02d548d6a58d9589a07b4ff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:20:21.836263 19558 raft_consensus.cc:1261] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Refusing update from remote peer c7c85cbd02d548d6a58d9589a07b4ff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:20:21.837232 19930 consensus_queue.cc:1035] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:20:21.837646 19924 consensus_queue.cc:1035] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:20:21.849380 19933 mvcc.cc:205] Tried to move back new op lower bound from 7315872243030245376 to 7315872242676461568. Current Snapshot: MvccSnapshot[applied={T|T < 7315872243030245376}]
I20260807 11:20:21.853435 19934 mvcc.cc:205] Tried to move back new op lower bound from 7315872243030245376 to 7315872242676461568. Current Snapshot: MvccSnapshot[applied={T|T < 7315872243030245376}]
I20260807 11:20:21.864497 19403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:20:21.864821 19403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:20:21.868687 19403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Sending DeleteTablet for 3 replicas of tablet 3892ff3a3131478384eeb5c1d435bd04
I20260807 11:20:21.869931 19537 tablet_service.cc:1514] Processing DeleteTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:21 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:49130
I20260807 11:20:21.870275 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19469
I20260807 11:20:21.870286 19683 tablet_service.cc:1514] Processing DeleteTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:21 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:33128
I20260807 11:20:21.871170 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 3892ff3a3131478384eeb5c1d435bd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:21 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60680
I20260807 11:20:21.881016 19946 tablet_replica.cc:330] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: stopping tablet replica
I20260807 11:20:21.882341 19944 tablet_replica.cc:330] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: stopping tablet replica
I20260807 11:20:21.882365 19946 raft_consensus.cc:2229] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:20:21.882687 19943 tablet_replica.cc:330] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: stopping tablet replica
I20260807 11:20:21.882788 19944 raft_consensus.cc:2229] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:20:21.883176 19943 raft_consensus.cc:2229] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:20:21.883359 19944 raft_consensus.cc:2258] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:21.883406 19946 raft_consensus.cc:2258] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:21.883783 19943 raft_consensus.cc:2258] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:21.886080 19946 ts_tablet_manager.cc:1905] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:20:21.886072 19943 ts_tablet_manager.cc:1905] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:20:21.893571 19943 ts_tablet_manager.cc:1918] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:21.893590 19946 ts_tablet_manager.cc:1918] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:21.893828 19946 log.cc:1196] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/3892ff3a3131478384eeb5c1d435bd04
I20260807 11:20:21.893944 19943 log.cc:1196] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/3892ff3a3131478384eeb5c1d435bd04
I20260807 11:20:21.894541 19946 ts_tablet_manager.cc:1939] T 3892ff3a3131478384eeb5c1d435bd04 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting consensus metadata
I20260807 11:20:21.894533 19943 ts_tablet_manager.cc:1939] T 3892ff3a3131478384eeb5c1d435bd04 P c74da024cd1349d99078a6291163f9b3: Deleting consensus metadata
I20260807 11:20:21.895972 19944 ts_tablet_manager.cc:1905] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:20:21 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:20:21.896517 19389 catalog_manager.cc:4841] TS c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749): tablet 3892ff3a3131478384eeb5c1d435bd04 (table test-table [id=6e04c403cdb54de094c5cb772aff8b2e]) successfully deleted
I20260807 11:20:21.896620 19390 catalog_manager.cc:4841] TS c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739): tablet 3892ff3a3131478384eeb5c1d435bd04 (table test-table [id=6e04c403cdb54de094c5cb772aff8b2e]) successfully deleted
I20260807 11:20:21.901908 19944 ts_tablet_manager.cc:1918] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:21.902096 19944 log.cc:1196] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/3892ff3a3131478384eeb5c1d435bd04
I20260807 11:20:21.902644 19944 ts_tablet_manager.cc:1939] T 3892ff3a3131478384eeb5c1d435bd04 P d313f5cbb893410a97819e969c03974b: Deleting consensus metadata
I20260807 11:20:21.904497 19389 catalog_manager.cc:4841] TS d313f5cbb893410a97819e969c03974b (127.2.223.130:42091): tablet 3892ff3a3131478384eeb5c1d435bd04 (table test-table [id=6e04c403cdb54de094c5cb772aff8b2e]) successfully deleted
I20260807 11:20:21.905836 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:21.871485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.2.223.129:45739)
0807 11:20:21.871708 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:21.871715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:21.884324 (+ 12609us) server_negotiation.cc:407] Connection header received
0807 11:20:21.884844 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:21.884850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:21.884939 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:21.885038 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:21.887293 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.888447 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.890632 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:21.891316 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:21.891422 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:21.897858 (+ 6436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:21.897873 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:21.897892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:21.897944 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:21.902038 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.903120 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.903131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.903134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.903336 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:21.903815 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:21.903820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:21.903823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:21.904314 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:21.904616 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:21.905195 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 11:20:21.905457 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:20:21.906636 19602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:21.998747 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19469
I20260807 11:20:22.028908 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19614
Aug 07 11:20:22 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:20:22.046514 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:22.029748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.2.223.130:42091)
0807 11:20:22.029925 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:22.029930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:22.030006 (+ 76us) server_negotiation.cc:407] Connection header received
0807 11:20:22.030190 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:22.030194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:22.030267 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:22.030352 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:22.032131 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:22.032827 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:22.034810 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:22.035225 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:22.035318 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:22.040646 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:22.040660 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:22.040676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:22.040708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:22.043624 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:22.044336 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:22.044343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:22.044346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:22.044481 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:22.044899 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:22.044904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:22.044906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:22.045276 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:22.045577 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:22.045910 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 11:20:22.046134 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:20:22.047247 19748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:22.141669 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19614
I20260807 11:20:22.163720 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19761
Aug 07 11:20:22 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:20:22.182238 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:22.164630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.2.223.131:33749)
0807 11:20:22.164843 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:22.164849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:22.164874 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:20:22.165016 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:22.165023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:22.165126 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:22.165251 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:22.166649 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:22.167812 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:22.169189 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:22.169911 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:22.170064 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:22.175017 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:22.175031 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:22.175053 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:22.175109 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:22.179313 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:22.179971 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:22.179981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:22.179985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:22.180145 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:22.180505 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:22.180510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:22.180514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:22.181090 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:22.181396 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:22.181591 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 11:20:22.181830 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:20:22.183073 19890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:22.308939 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19761
I20260807 11:20:22.333214 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19371
I20260807 11:20:22.334403 19435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:22.450855 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19371
I20260807 11:20:42.479408 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--webserver_interface=127.2.223.190
--webserver_port=45773
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:20:42.844632 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:42.844913 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:42.845082 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:42.855756 19951 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:20:42.855890 19951 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:20:42.855968 19951 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:20:42.856055 19951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:20:42.856153 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:42.856276 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:42.856345 19951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:20:42.856426 19951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:20:42.867013 19951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:42.868379 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:42.869817 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:42.897089 19959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:42.897010 19962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:42.897176 19958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:42.897346 19951 server_base.cc:1034] running on GCE node
I20260807 11:20:42.898573 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:42.900241 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:42.901499 19951 hybrid_clock.cc:648] HybridClock initialized: now 1786101642901422 us; error 51 us; skew 500 ppm
I20260807 11:20:42.901902 19951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:42 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101642, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:42.905728 19951 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:20:42.907434 19951 webserver.cc:460] Webserver started at http://127.2.223.190:45773/ using document root <none> and password file <none>
I20260807 11:20:42.908180 19951 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:42.908288 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:42.914570 19951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:20:42.917454 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:42.918946 19951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:20:42.919165 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "77f8804e53784608bb90c7bd7101bde0"
format_stamp: "Formatted at 2026-08-07 11:20:19 on dist-test-slave-5l3k"
server_key: "62341a03e5cd7c4530232296d48b64bb"
server_key_iv: "abc9515059a319d0d0ba3fc0b62573fb"
server_key_version: "encryptionkey@0"
I20260807 11:20:42.920305 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:42.944545 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:42.948899 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:42.949213 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:42.969709 19951 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:45069
I20260807 11:20:42.969740 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:45069 every 8 connection(s)
I20260807 11:20:42.971318 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:20:42.974761 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 19951
I20260807 11:20:42.975677 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:45739
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=32905
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:20:42.978883 20026 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:20:42.982216 20026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap starting.
I20260807 11:20:43.002632 20026 log.cc:824] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:43.015581 20026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:20:43.016149 20026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap complete.
I20260807 11:20:43.026230 20026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:43.027213 20026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Initialized, Role: FOLLOWER
I20260807 11:20:43.027789 20026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:43.028044 20026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:20:43.028177 20026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:20:43.028357 20026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:20:43.032150 20026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:43.032660 20026 leader_election.cc:302] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f8804e53784608bb90c7bd7101bde0; no voters:
I20260807 11:20:43.033295 20026 leader_election.cc:288] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:20:43.033429 20030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:20:43.034814 20030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 LEADER]: Becoming Leader. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Running, Role: LEADER
I20260807 11:20:43.035480 20030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:20:43.035737 20026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:20:43.039798 20032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f8804e53784608bb90c7bd7101bde0. Latest consensus state: current_term: 2 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:20:43.039860 20031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:20:43.040282 20032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:43.040292 20031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:20:43.043583 20038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:20:43.053447 20038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:20:43.055542 20038 catalog_manager.cc:1259] Loaded cluster ID: d3defa5eee34452bb9986f088f88f83f
I20260807 11:20:43.055842 20038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:20:43.061259 20038 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:20:43.063315 20038 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Loaded TSK: 0
I20260807 11:20:43.124467 20053 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:20:43.125797 20038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Generated new TSK 1
I20260807 11:20:43.126185 20038 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:20:43.363783 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:43.364105 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:43.364266 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:43.371547 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:43.371699 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:43.371876 20028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:20:43.381865 20028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:45739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=32905
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:43.383255 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:43.384567 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:43.399631 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:43.400218 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:43.401003 20062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:43.401081 20028 server_base.cc:1034] running on GCE node
I20260807 11:20:43.401818 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:43.403082 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:43.404317 20028 hybrid_clock.cc:648] HybridClock initialized: now 1786101643404254 us; error 57 us; skew 500 ppm
I20260807 11:20:43.404644 20028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:43 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:43.407979 20028 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:20:43.409627 20028 webserver.cc:460] Webserver started at http://127.2.223.129:32905/ using document root <none> and password file <none>
I20260807 11:20:43.410329 20028 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:43.410481 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:43.415970 20028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:20:43.418898 20069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.420390 20028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:20:43.420565 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "c74da024cd1349d99078a6291163f9b3"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "3a3d50444babdca9456c76a0b440f4d8"
server_key_iv: "739c0f32fc17fe62c629c94fc0da3c3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:43.421518 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:43.447379 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:43.451856 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:43.452155 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:43.453204 20028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:43.454658 20028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:43.454783 20028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.454909 20028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:43.455008 20028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.489570 20028 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:45739
I20260807 11:20:43.489665 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:45739 every 8 connection(s)
I20260807 11:20:43.491196 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:20:43.499632 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20028
I20260807 11:20:43.500486 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:42091
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=44819
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:20:43 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:43.519091 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:43.494131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:51737 (local address 127.2.223.190:45069)
0807 11:20:43.494883 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:43.494906 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:43.495482 (+ 576us) server_negotiation.cc:407] Connection header received
0807 11:20:43.497626 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:43.497640 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:43.497995 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:43.498319 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:43.500391 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.501760 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.504418 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.505183 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.505331 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:43.509941 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:43.509955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:43.509979 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:43.510033 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:43.514018 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:43.515166 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:43.515175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:43.515182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:43.515366 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:43.515923 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:43.515927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:43.515929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:43.516430 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:43.516589 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:43.517563 (+ 974us) server_negotiation.cc:299] Negotiation successful
0807 11:20:43.517919 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":316,"threads_started":1}
I20260807 11:20:43.519474 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:43.494474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:51737)
0807 11:20:43.495289 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:43.495333 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:43.497097 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:43.498475 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:43.498534 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:43.499261 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:43.500106 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:43.500139 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.501965 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.501974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:43.504224 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:43.504245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.505450 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.505460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:43.505800 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:43.506644 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:43.506691 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:43.509542 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:43.514229 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:43.514241 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:43.514261 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:43.514987 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:43.515499 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:43.515507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:43.515513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:43.515796 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:43.516584 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:43.516596 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:43.517048 (+ 452us) client_negotiation.cc:769] Sending connection context
0807 11:20:43.517376 (+ 328us) client_negotiation.cc:240] Negotiation successful
0807 11:20:43.518004 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260807 11:20:43.522660 20190 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:43.523010 20190 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:43.523761 20190 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:43.526280 19989 ts_manager.cc:194] Registered new tserver with Master: c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739)
I20260807 11:20:43.527869 19989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:51737
W20260807 11:20:43.868120 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:43.868479 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:43.868669 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:43.876575 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:43.876729 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:43.876904 20194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:20:43.886948 20194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:42091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=44819
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:43.888280 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:43.889653 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:43.903218 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:43.903182 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:43.903254 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:43.904412 20194 server_base.cc:1034] running on GCE node
I20260807 11:20:43.905130 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:43.906080 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:43.907310 20194 hybrid_clock.cc:648] HybridClock initialized: now 1786101643907272 us; error 37 us; skew 500 ppm
I20260807 11:20:43.907650 20194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:43 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:43.911041 20194 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:20:43.912436 20194 webserver.cc:460] Webserver started at http://127.2.223.130:44819/ using document root <none> and password file <none>
I20260807 11:20:43.913074 20194 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:43.913199 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:43.918582 20194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:20:43.921378 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.922770 20194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:20:43.922899 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "d313f5cbb893410a97819e969c03974b"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "e8fb548b269022ba45927168715256cf"
server_key_iv: "6aa61f5c2043c1ba2a70e6a009abf2d2"
server_key_version: "encryptionkey@0"
I20260807 11:20:43.923826 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:43.944738 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:43.948818 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:43.949055 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:43.949985 20194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:43.951372 20194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:43.951465 20194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.951543 20194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:43.951604 20194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:43.987936 20194 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42091
I20260807 11:20:43.987987 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42091 every 8 connection(s)
I20260807 11:20:43.989598 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:20:43.998970 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20194
I20260807 11:20:43.999712 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:33749
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=36081
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:20:43 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:44.010460 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:43.992390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:37341 (local address 127.2.223.190:45069)
0807 11:20:43.992620 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:43.992626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:43.993860 (+ 1234us) server_negotiation.cc:407] Connection header received
0807 11:20:43.995885 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:43.995890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:43.995997 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:43.996104 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:43.997527 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.998255 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.999994 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.000447 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.000532 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.004319 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.004325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.004328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.004364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.007237 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.008013 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.008024 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.008027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.008164 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.008618 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.008628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.008630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.008915 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.009078 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.009825 (+ 747us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.010066 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:20:44.011518 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:43.992849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:37341)
0807 11:20:43.993697 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:43.993742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:43.995592 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:43.996316 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:43.996323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:43.996823 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:43.997358 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:43.997375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:43.998403 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:43.998407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:43.999772 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:43.999810 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.000818 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.000823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.001029 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.001732 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:44.001761 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:44.004057 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:44.007386 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:44.007396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:44.007413 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:44.007870 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:44.008283 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:44.008289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:44.008292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:44.008491 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:44.009050 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:44.009059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:44.009427 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.009734 (+ 307us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.010359 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":377,"threads_started":1}
I20260807 11:20:44.013252 20328 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:44.013573 20328 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:44.014385 20328 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:44.016287 19989 ts_manager.cc:194] Registered new tserver with Master: d313f5cbb893410a97819e969c03974b (127.2.223.130:42091)
I20260807 11:20:44.017719 19989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:37341
W20260807 11:20:44.359716 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:20:44.359982 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:20:44.360083 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:20:44.367115 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:20:44.367235 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:20:44.367352 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:20:44.377085 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:33749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=36081
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:20:44.378319 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:20:44.379606 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:20:44.392582 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:44.392545 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:20:44.392628 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:20:44.393637 20331 server_base.cc:1034] running on GCE node
I20260807 11:20:44.394440 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:20:44.395516 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:20:44.396764 20331 hybrid_clock.cc:648] HybridClock initialized: now 1786101644396703 us; error 35 us; skew 500 ppm
I20260807 11:20:44.397140 20331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:20:44 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:20:44.400442 20331 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:20:44.401940 20331 webserver.cc:460] Webserver started at http://127.2.223.131:36081/ using document root <none> and password file <none>
I20260807 11:20:44.402693 20331 fs_manager.cc:359] Metadata directory not provided
I20260807 11:20:44.402863 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:20:44.408216 20331 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260807 11:20:44.411387 20347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:44.412820 20331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:20:44.413005 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
format_stamp: "Formatted at 2026-08-07 11:20:21 on dist-test-slave-5l3k"
server_key: "52e59692f9002263e890951cfe3d7797"
server_key_iv: "2bf8462ae3a73c97a28d46789228ad3f"
server_key_version: "encryptionkey@0"
I20260807 11:20:44.414047 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:20:44.451136 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:20:44.455521 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:20:44.455818 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:20:44.456838 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:20:44.458235 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:20:44.458366 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:20:44.458521 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:20:44.458618 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 11:20:44.495384 20331 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:33749
I20260807 11:20:44.495473 20467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:33749 every 8 connection(s)
I20260807 11:20:44.496937 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:20:44.497996 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20331
Aug 07 11:20:44 dist-test-slave-5l3k krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101644, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:20:44.516847 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.499240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.2.223.190:45069)
0807 11:20:44.499449 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.499457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.500030 (+ 573us) server_negotiation.cc:407] Connection header received
0807 11:20:44.500343 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.500355 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.500422 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.500532 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:44.503004 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.503788 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.506058 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.506788 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.506916 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.510294 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.510301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.510306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.510349 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.513770 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.514645 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.514651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.514654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.514809 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.515189 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.515196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.515200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.515583 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.515901 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.516170 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.516426 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:20:44.521229 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.499793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:52099 (local address 127.2.223.190:45069)
0807 11:20:44.500281 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.500288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.501590 (+ 1302us) server_negotiation.cc:407] Connection header received
0807 11:20:44.504033 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.504038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.504117 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.504215 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:44.506192 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.507301 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.509844 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.510372 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.510570 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.514709 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.514715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.514723 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.514760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.518126 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.519033 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.519038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.519041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.519176 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.519610 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.519663 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.519669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.520046 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.520157 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.520812 (+ 655us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.520956 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260807 11:20:44.522197 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.500502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:52099)
0807 11:20:44.501396 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.501457 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.503652 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.504506 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.504518 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.505247 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:20:44.506013 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.506040 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.507486 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.507491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.509701 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.509718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.510651 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.510657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.510925 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.511690 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:20:44.511717 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:20:44.514421 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:20:44.518264 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:44.518270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:44.518303 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:44.518920 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:44.519286 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:20:44.519289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:20:44.519292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:20:44.519469 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:20:44.520185 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:20:44.520191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:20:44.520486 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.520714 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.521074 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":312,"threads_started":1}
I20260807 11:20:44.523670 20468 heartbeater.cc:347] Connected to a master server at 127.2.223.190:45069
I20260807 11:20:44.524000 20468 heartbeater.cc:464] Registering TS with master...
I20260807 11:20:44.524827 20468 heartbeater.cc:511] Master 127.2.223.190:45069 requested a full tablet report, sending...
I20260807 11:20:44.526255 19989 ts_manager.cc:194] Registered new tserver with Master: c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749)
I20260807 11:20:44.527093 19989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:52099
I20260807 11:20:44.531090 20190 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:44.532524 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:20:44.539919 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.533531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.2.223.190:45069)
0807 11:20:44.533686 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.533690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.534175 (+ 485us) server_negotiation.cc:407] Connection header received
0807 11:20:44.534403 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.534408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.534616 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.534758 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:20:44.535185 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.536001 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.537595 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.538198 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.538330 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.538986 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:20:44.539262 (+ 276us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 11:20:44.539651 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52488: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 11:20:44.540067 20472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52488: token expired
W20260807 11:20:44.540726 19903 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.223.190:45069)
I20260807 11:20:44.554071 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.541517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.2.223.190:45069)
0807 11:20:44.541692 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.541697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.541719 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:20:44.541755 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.541758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.541806 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.541908 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:44.543341 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.544063 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.545395 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.545808 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.545938 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.548801 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.548808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.548812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.548852 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.551365 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.552091 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.552097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.552100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.552253 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.552634 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.552640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.552644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.552942 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.553252 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.553531 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.553753 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:20:44.557701 19987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:20:44.565079 19987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:20:44.598130 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.577001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33749 (local address 127.0.0.1:60034)
0807 11:20:44.578267 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.578329 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.578616 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.589529 (+ 10913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.589538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.589574 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:44.589935 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.589955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.592073 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.592080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.595541 (+ 3461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.595553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.596933 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.596939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.597347 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.597403 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.597543 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.597753 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":866,"threads_started":1}
I20260807 11:20:44.598171 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.577388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.2.223.131:33749)
0807 11:20:44.588539 (+ 11151us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.588551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.588593 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:20:44.588714 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.588725 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.589025 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.589264 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:44.590184 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.591929 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.595714 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.596700 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.596867 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.597120 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.597606 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.597794 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10931,"thread_start_us":11132,"threads_started":1}
I20260807 11:20:44.599552 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.579282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.2.223.130:42091)
0807 11:20:44.589215 (+ 9933us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.589224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.591155 (+ 1931us) server_negotiation.cc:407] Connection header received
0807 11:20:44.591323 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.591331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.591532 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.591718 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:44.592588 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.593554 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.596823 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.598223 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.598362 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.598587 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.599096 (+ 509us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.599252 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9747,"thread_start_us":9933,"threads_started":1}
I20260807 11:20:44.599643 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.577524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60488)
0807 11:20:44.588199 (+ 10675us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.588228 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.588398 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.591383 (+ 2985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.591388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.591418 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:44.591794 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.591805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.593806 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.593813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.596642 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.596658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.598781 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.598788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.598941 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.598964 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.599069 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.599207 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10478,"mutex_wait_us":110,"thread_start_us":13186,"threads_started":1}
I20260807 11:20:44.599864 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.578685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:42091 (local address 127.0.0.1:37356)
0807 11:20:44.590999 (+ 12314us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.591020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.591159 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.591944 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.591949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.591979 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:44.592414 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.592423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.593702 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.593708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.596642 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.596658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.598694 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.598702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.598886 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.598918 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.599023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.599142 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12195,"mutex_wait_us":137,"thread_start_us":9108,"threads_started":1}
I20260807 11:20:44.600664 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.578207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.2.223.129:45739)
0807 11:20:44.590652 (+ 12445us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.590664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.590701 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:20:44.590835 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.590844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.591109 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.591410 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:44.591966 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.593599 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.596835 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.598467 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.599597 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.600047 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.600129 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.600322 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12198,"thread_start_us":12043,"threads_started":1}
I20260807 11:20:44.602805 20398 tablet_service.cc:1467] Processing CreateTablet for tablet cc4ac635ef39449586b437bc348c9a43 (DEFAULT_TABLE table=test-table [id=88ca4285f4b644a6b860342a0d1019bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:44.603693 20259 tablet_service.cc:1467] Processing CreateTablet for tablet cc4ac635ef39449586b437bc348c9a43 (DEFAULT_TABLE table=test-table [id=88ca4285f4b644a6b860342a0d1019bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:44.603879 20120 tablet_service.cc:1467] Processing CreateTablet for tablet cc4ac635ef39449586b437bc348c9a43 (DEFAULT_TABLE table=test-table [id=88ca4285f4b644a6b860342a0d1019bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:20:44.604418 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4ac635ef39449586b437bc348c9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:44.605037 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4ac635ef39449586b437bc348c9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:44.605058 20120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4ac635ef39449586b437bc348c9a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:20:44.618245 20486 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Bootstrap starting.
I20260807 11:20:44.621470 20486 tablet_bootstrap.cc:654] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:44.622597 20486 log.cc:824] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:44.624310 20487 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Bootstrap starting.
I20260807 11:20:44.623548 20488 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Bootstrap starting.
I20260807 11:20:44.626767 20486 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: No bootstrap required, opened a new log
I20260807 11:20:44.627067 20486 ts_tablet_manager.cc:1397] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 11:20:44.628044 20487 tablet_bootstrap.cc:654] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:44.629071 20488 tablet_bootstrap.cc:654] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Neither blocks nor log segments found. Creating new log.
I20260807 11:20:44.629344 20487 log.cc:824] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:44.630313 20488 log.cc:824] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:20:44.632531 20487 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: No bootstrap required, opened a new log
I20260807 11:20:44.632825 20487 ts_tablet_manager.cc:1397] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:20:44.633438 20488 tablet_bootstrap.cc:492] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: No bootstrap required, opened a new log
I20260807 11:20:44.633702 20488 ts_tablet_manager.cc:1397] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 11:20:44.635840 20486 raft_consensus.cc:348] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.636256 20486 raft_consensus.cc:374] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:44.636390 20486 raft_consensus.cc:729] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74da024cd1349d99078a6291163f9b3, State: Initialized, Role: FOLLOWER
I20260807 11:20:44.637051 20486 consensus_queue.cc:260] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.638715 20486 ts_tablet_manager.cc:1428] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260807 11:20:44.644327 20488 raft_consensus.cc:348] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.644788 20488 raft_consensus.cc:374] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:44.644893 20488 raft_consensus.cc:729] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d313f5cbb893410a97819e969c03974b, State: Initialized, Role: FOLLOWER
I20260807 11:20:44.645495 20488 consensus_queue.cc:260] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.645637 20487 raft_consensus.cc:348] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.646091 20487 raft_consensus.cc:374] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:20:44.646229 20487 raft_consensus.cc:729] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c85cbd02d548d6a58d9589a07b4ff3, State: Initialized, Role: FOLLOWER
I20260807 11:20:44.646713 20328 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:44.646811 20487 consensus_queue.cc:260] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.647508 20488 ts_tablet_manager.cc:1428] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 11:20:44.648006 20468 heartbeater.cc:503] Master 127.2.223.190:45069 was elected leader, sending a full tablet report...
I20260807 11:20:44.648550 20487 ts_tablet_manager.cc:1428] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260807 11:20:44.653195 20492 raft_consensus.cc:482] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:20:44.653468 20492 raft_consensus.cc:504] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.655233 20492 leader_election.cc:288] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749), d313f5cbb893410a97819e969c03974b (127.2.223.130:42091)
I20260807 11:20:44.664351 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.656555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:53401 (local address 127.2.223.130:42091)
0807 11:20:44.656760 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.656765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.656988 (+ 223us) server_negotiation.cc:407] Connection header received
0807 11:20:44.657120 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.657123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.657217 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.657310 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:44.657990 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.658879 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.661731 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.663046 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.663196 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.663278 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.663927 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.664068 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:20:44.664351 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.655574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:49251 (local address 127.2.223.131:33749)
0807 11:20:44.655771 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.655779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.656312 (+ 533us) server_negotiation.cc:407] Connection header received
0807 11:20:44.656636 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.656643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.656724 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.656794 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:20:44.657643 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.658719 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.661731 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.663030 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.663207 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.663278 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.663927 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.664093 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 11:20:44.664724 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.655589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33749 (local address 127.2.223.129:49251)
0807 11:20:44.656210 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.656231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.656438 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.657022 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.657028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.657050 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:44.657450 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.657463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.658865 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.658869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.661540 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.661555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.663374 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.663382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.663627 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.663770 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.663875 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.664068 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":288,"thread_start_us":305,"threads_started":1}
I20260807 11:20:44.664757 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.656245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:42091 (local address 127.2.223.129:53401)
0807 11:20:44.656889 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.656907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:20:44.657023 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:20:44.657354 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:20:44.657361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:20:44.657397 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:20:44.657815 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.657828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.659003 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.659009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.661540 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:20:44.661555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.663389 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.663394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:20:44.663627 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.663767 (+ 140us) client_negotiation.cc:769] Sending connection context
0807 11:20:44.663875 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 11:20:44.664068 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":169,"thread_start_us":338,"threads_started":1}
I20260807 11:20:44.665861 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4ac635ef39449586b437bc348c9a43" candidate_uuid: "c74da024cd1349d99078a6291163f9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d313f5cbb893410a97819e969c03974b" is_pre_election: true
I20260807 11:20:44.665882 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4ac635ef39449586b437bc348c9a43" candidate_uuid: "c74da024cd1349d99078a6291163f9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" is_pre_election: true
I20260807 11:20:44.666373 20281 raft_consensus.cc:2454] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74da024cd1349d99078a6291163f9b3 in term 0.
I20260807 11:20:44.666383 20422 raft_consensus.cc:2454] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74da024cd1349d99078a6291163f9b3 in term 0.
I20260807 11:20:44.667084 20071 leader_election.cc:302] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c74da024cd1349d99078a6291163f9b3, c7c85cbd02d548d6a58d9589a07b4ff3; no voters:
I20260807 11:20:44.667482 20492 raft_consensus.cc:2781] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:20:44.667650 20492 raft_consensus.cc:482] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:20:44.667764 20492 raft_consensus.cc:3037] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:44.669981 20492 raft_consensus.cc:504] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.670615 20492 leader_election.cc:288] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [CANDIDATE]: Term 1 election: Requested vote from peers c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749), d313f5cbb893410a97819e969c03974b (127.2.223.130:42091)
I20260807 11:20:44.671126 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4ac635ef39449586b437bc348c9a43" candidate_uuid: "c74da024cd1349d99078a6291163f9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
I20260807 11:20:44.671181 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4ac635ef39449586b437bc348c9a43" candidate_uuid: "c74da024cd1349d99078a6291163f9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d313f5cbb893410a97819e969c03974b"
I20260807 11:20:44.671321 20422 raft_consensus.cc:3037] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:44.671450 20281 raft_consensus.cc:3037] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:20:44.673668 20422 raft_consensus.cc:2454] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74da024cd1349d99078a6291163f9b3 in term 1.
I20260807 11:20:44.673782 20281 raft_consensus.cc:2454] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74da024cd1349d99078a6291163f9b3 in term 1.
I20260807 11:20:44.674109 20071 leader_election.cc:302] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c74da024cd1349d99078a6291163f9b3, c7c85cbd02d548d6a58d9589a07b4ff3; no voters:
I20260807 11:20:44.674526 20492 raft_consensus.cc:2781] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:20:44.675099 20492 raft_consensus.cc:686] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 1 LEADER]: Becoming Leader. State: Replica: c74da024cd1349d99078a6291163f9b3, State: Running, Role: LEADER
I20260807 11:20:44.675597 20492 consensus_queue.cc:237] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } }
I20260807 11:20:44.680435 19987 catalog_manager.cc:5495] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c74da024cd1349d99078a6291163f9b3 (127.2.223.129). New cstate: current_term: 1 leader_uuid: "c74da024cd1349d99078a6291163f9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74da024cd1349d99078a6291163f9b3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 45739 } health_report { overall_health: HEALTHY } } }
I20260807 11:20:44.714769 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.708991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.2.223.129:45739)
0807 11:20:44.709264 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.709272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.709380 (+ 108us) server_negotiation.cc:407] Connection header received
0807 11:20:44.709662 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.709666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.709747 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.709921 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:20:44.710359 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.711156 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.712589 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.713016 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.713160 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.713672 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:20:44.714173 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:20:44.714339 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.714399 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.714479 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:20:44.723179 20281 raft_consensus.cc:1261] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Refusing update from remote peer c74da024cd1349d99078a6291163f9b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:20:44.723299 20422 raft_consensus.cc:1261] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Refusing update from remote peer c74da024cd1349d99078a6291163f9b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:20:44.724253 20498 consensus_queue.cc:1035] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d313f5cbb893410a97819e969c03974b" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:20:44.724694 20492 consensus_queue.cc:1035] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c85cbd02d548d6a58d9589a07b4ff3" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:20:44.735205 20501 mvcc.cc:205] Tried to move back new op lower bound from 7315872336775405568 to 7315872336595181568. Current Snapshot: MvccSnapshot[applied={T|T < 7315872336775405568}]
W20260807 11:20:44.741765 20329 tablet.cc:2369] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:20:44.751600 19987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:20:44.751924 19987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:20:44.756070 19987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Sending DeleteTablet for 3 replicas of tablet cc4ac635ef39449586b437bc348c9a43
I20260807 11:20:44.757280 20398 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4ac635ef39449586b437bc348c9a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60034
I20260807 11:20:44.757418 20259 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4ac635ef39449586b437bc348c9a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37356
I20260807 11:20:44.757442 20258 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4ac635ef39449586b437bc348c9a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37356
I20260807 11:20:44.758200 20120 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4ac635ef39449586b437bc348c9a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:20:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60488
I20260807 11:20:44.758339 20514 tablet_replica.cc:330] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: stopping tablet replica
I20260807 11:20:44.758771 20514 raft_consensus.cc:2229] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:20:44.759182 20514 raft_consensus.cc:2258] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:44.759541 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20028
I20260807 11:20:44.769496 20515 tablet_replica.cc:330] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: stopping tablet replica
I20260807 11:20:44.770105 20515 raft_consensus.cc:2229] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:20:44.770993 20515 raft_consensus.cc:2258] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:44.773701 20515 ts_tablet_manager.cc:1905] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:20:44.781317 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.761148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.2.223.129:45739)
0807 11:20:44.761308 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.761314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.761331 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:20:44.761383 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.761388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.761477 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.761573 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:44.763725 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.765099 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.767189 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.767920 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.768054 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.772312 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.772331 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.772349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.772394 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.776854 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.777890 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.777901 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.777907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.778112 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.778606 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.778612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.778615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.779085 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.779429 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.780669 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.780928 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:20:44.783200 20514 ts_tablet_manager.cc:1905] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:20:44.783951 20185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:44.784936 20513 tablet_replica.cc:330] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: stopping tablet replica
I20260807 11:20:44.785346 20513 raft_consensus.cc:2229] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:20:44.785821 20513 raft_consensus.cc:2258] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:20:44.787849 20513 ts_tablet_manager.cc:1905] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:20:44.792982 20514 ts_tablet_manager.cc:1918] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:44.793182 20514 log.cc:1196] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/cc4ac635ef39449586b437bc348c9a43
I20260807 11:20:44.793684 20514 ts_tablet_manager.cc:1939] T cc4ac635ef39449586b437bc348c9a43 P d313f5cbb893410a97819e969c03974b: Deleting consensus metadata
I20260807 11:20:44.794265 20513 ts_tablet_manager.cc:1918] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:44.794430 20513 log.cc:1196] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/cc4ac635ef39449586b437bc348c9a43
I20260807 11:20:44.794967 20513 ts_tablet_manager.cc:1939] T cc4ac635ef39449586b437bc348c9a43 P c7c85cbd02d548d6a58d9589a07b4ff3: Deleting consensus metadata
I20260807 11:20:44.795545 19971 catalog_manager.cc:4841] TS d313f5cbb893410a97819e969c03974b (127.2.223.130:42091): tablet cc4ac635ef39449586b437bc348c9a43 (table test-table [id=88ca4285f4b644a6b860342a0d1019bb]) successfully deleted
W20260807 11:20:44.796190 19971 catalog_manager.cc:4808] TS d313f5cbb893410a97819e969c03974b (127.2.223.130:42091): delete failed for tablet cc4ac635ef39449586b437bc348c9a43 because the tablet was not found. No further retry: Not found: Tablet not found: cc4ac635ef39449586b437bc348c9a43
I20260807 11:20:44.796509 19971 catalog_manager.cc:4841] TS c7c85cbd02d548d6a58d9589a07b4ff3 (127.2.223.131:33749): tablet cc4ac635ef39449586b437bc348c9a43 (table test-table [id=88ca4285f4b644a6b860342a0d1019bb]) successfully deleted
I20260807 11:20:44.899369 20515 ts_tablet_manager.cc:1918] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:20:44.899699 20515 log.cc:1196] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/cc4ac635ef39449586b437bc348c9a43
I20260807 11:20:44.900184 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20028
W20260807 11:20:44.923877 19972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.223.129:45739 (error 108)
I20260807 11:20:44.924713 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20194
I20260807 11:20:44.925984 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.925488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60528)
0807 11:20:44.925745 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.925847 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260807 11:20:44.926390 19972 catalog_manager.cc:4584] TS c74da024cd1349d99078a6291163f9b3 (127.2.223.129:45739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc4ac635ef39449586b437bc348c9a43: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
I20260807 11:20:44.939419 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.925573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.2.223.130:42091)
0807 11:20:44.925778 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:44.925784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:44.925805 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:20:44.925936 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:44.925941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:44.926040 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:44.926165 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:44.927958 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.928810 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.930056 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:44.930446 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:44.930635 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:44.933630 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:44.933644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:44.933657 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:44.933687 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:44.936479 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.937306 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.937313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.937316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.937456 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:44.937810 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:44.937813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:44.937815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:44.938164 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:44.938388 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:44.938859 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 11:20:44.939059 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:20:44.940182 20323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:44.980154 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.979574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60530)
0807 11:20:44.979825 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.979952 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 11:20:44.981355 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:44.980934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60542)
0807 11:20:44.981100 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:44.981208 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 11:20:45.034325 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20194
I20260807 11:20:45.042239 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.041732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60550)
0807 11:20:45.041962 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:45.042088 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:20:45.043427 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.043043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60554)
0807 11:20:45.043225 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:45.043308 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 11:20:45.056242 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20331
I20260807 11:20:45.069404 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.057063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.2.223.131:33749)
0807 11:20:45.057242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 11:20:45.057247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:20:45.057268 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:20:45.057383 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:20:45.057388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:20:45.057451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:20:45.057547 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:20:45.058962 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:45.059701 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:45.060994 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:20:45.061377 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:20:45.061465 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:20:45.064231 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:20:45.064245 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:20:45.064263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:20:45.064295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:20:45.067017 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:45.067674 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:45.067680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:45.067682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:45.067808 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:20:45.068121 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:20:45.068124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:20:45.068135 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:20:45.068482 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:20:45.068699 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 11:20:45.068965 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 11:20:45.069139 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:20:45.070056 20463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:45.124104 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.123530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60558)
0807 11:20:45.123786 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:45.123906 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 11:20:45.125447 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.124987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60568)
0807 11:20:45.125229 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:45.125322 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 11:20:45.162885 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20331
I20260807 11:20:45.185911 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 19951
I20260807 11:20:45.187110 20021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:20:45.305218 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:20:45.304307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:45739 (local address 127.0.0.1:60584)
0807 11:20:45.304542 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 11:20:45.304703 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 11:20:45.305546 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 19951
I20260807 11:21:05.332916 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--webserver_interface=127.2.223.190
--webserver_port=45773
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:05.715662 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:05.716055 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:05.716226 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:05.723644 20522 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:21:05.723798 20522 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:21:05.723896 20522 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:21:05.724000 20522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:05.724102 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:05.724207 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:05.724299 20522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:05.724381 20522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:05.734494 20522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:05.735919 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:05.737353 20522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:05.749810 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:05.749888 20529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:05.749861 20533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:05.750649 20522 server_base.cc:1034] running on GCE node
I20260807 11:21:05.751564 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:05.753139 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:05.754406 20522 hybrid_clock.cc:648] HybridClock initialized: now 1786101665754346 us; error 57 us; skew 500 ppm
I20260807 11:21:05.754791 20522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:05 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101665, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:05.758373 20522 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:21:05.760288 20522 webserver.cc:460] Webserver started at http://127.2.223.190:45773/ using document root <none> and password file <none>
I20260807 11:21:05.761082 20522 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:05.761240 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:05.767243 20522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:21:05.770900 20540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:05.772272 20522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:21:05.772482 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "77f8804e53784608bb90c7bd7101bde0"
format_stamp: "Formatted at 2026-08-07 11:20:19 on dist-test-slave-5l3k"
server_key: "62341a03e5cd7c4530232296d48b64bb"
server_key_iv: "abc9515059a319d0d0ba3fc0b62573fb"
server_key_version: "encryptionkey@0"
I20260807 11:21:05.773579 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:05.792891 20522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:05.797828 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:05.798192 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:05.818902 20522 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:45069
I20260807 11:21:05.818938 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:45069 every 8 connection(s)
I20260807 11:21:05.820557 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:05.824963 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20522
I20260807 11:21:05.826144 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:45739
--local_ip_for_outbound_sockets=127.2.223.129
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=32905
--webserver_interface=127.2.223.129
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:21:05.829432 20597 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:21:05.834862 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap starting.
I20260807 11:21:05.858723 20597 log.cc:824] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:05.876250 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:21:05.876906 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Bootstrap complete.
I20260807 11:21:05.887063 20597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:21:05.888051 20597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Initialized, Role: FOLLOWER
I20260807 11:21:05.888799 20597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:21:05.889102 20597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:05.889251 20597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:05.889472 20597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:21:05.893868 20597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:21:05.894529 20597 leader_election.cc:302] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f8804e53784608bb90c7bd7101bde0; no voters:
I20260807 11:21:05.895330 20597 leader_election.cc:288] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 11:21:05.895495 20601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:21:05.896780 20601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [term 3 LEADER]: Becoming Leader. State: Replica: 77f8804e53784608bb90c7bd7101bde0, State: Running, Role: LEADER
I20260807 11:21:05.897400 20601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } }
I20260807 11:21:05.897768 20597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:05.900056 20603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f8804e53784608bb90c7bd7101bde0. Latest consensus state: current_term: 3 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:21:05.900532 20603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:05.901175 20602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "77f8804e53784608bb90c7bd7101bde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f8804e53784608bb90c7bd7101bde0" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 45069 } } }
I20260807 11:21:05.901525 20602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:05.905351 20607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:05.916993 20607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:05.919209 20607 catalog_manager.cc:1259] Loaded cluster ID: d3defa5eee34452bb9986f088f88f83f
I20260807 11:21:05.919358 20607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:05.924965 20607 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:05.927250 20607 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Loaded TSK: 0
I20260807 11:21:05.927443 20607 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77f8804e53784608bb90c7bd7101bde0: Loaded TSK: 1
F20260807 11:21:05.928892 20607 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7b20c1f9ad google::LogMessage::Fail() at ??:0
@ 0x7f7b20c2288a google::LogMessage::SendToLog() at ??:0
@ 0x7f7b20c1f4de google::LogMessage::Flush() at ??:0
@ 0x7f7b20c20a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7b249837d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7b2497273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7b3664e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7b3664d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7b366d92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7b366d90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7b367235e9 std::function<>::operator()() at ??:0
@ 0x7f7b36651cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7b3664fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7b366d7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7b366d7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b2386b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7b2386e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7b2386e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b23849eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7b26b236db start_thread at ??:0
@ 0x7f7b1fab961f clone at ??:0
W20260807 11:21:06.278401 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:06.278792 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:06.279000 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:06.286625 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:06.286773 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:06.286971 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:06.297323 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:45739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=32905
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:06.298805 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:06.300225 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:06.313910 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:06.313956 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:06.314007 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:06.314906 20599 server_base.cc:1034] running on GCE node
I20260807 11:21:06.315563 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:06.316560 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:06.317777 20599 hybrid_clock.cc:648] HybridClock initialized: now 1786101666317729 us; error 35 us; skew 500 ppm
I20260807 11:21:06.318112 20599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:06 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:06.321491 20599 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:21:06.323077 20599 webserver.cc:460] Webserver started at http://127.2.223.129:32905/ using document root <none> and password file <none>
I20260807 11:21:06.323765 20599 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:06.323873 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:06.329385 20599 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:21:06.332175 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:06.333535 20599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:21:06.333714 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "c74da024cd1349d99078a6291163f9b3"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "3a3d50444babdca9456c76a0b440f4d8"
server_key_iv: "739c0f32fc17fe62c629c94fc0da3c3f"
server_key_version: "encryptionkey@0"
I20260807 11:21:06.334816 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:06.357237 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:06.361490 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:06.361779 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:06.362985 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:06.364830 20666 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260807 11:21:06.370281 20666 ts_tablet_manager.cc:1852] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 11:21:06.371340 20666 ts_tablet_manager.cc:1905] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:06.378822 20666 ts_tablet_manager.cc:1918] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:06.379129 20666 ts_tablet_manager.cc:1939] T cc4ac635ef39449586b437bc348c9a43 P c74da024cd1349d99078a6291163f9b3: Deleting consensus metadata
I20260807 11:21:06.379639 20599 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 11:21:06.379767 20599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260807 11:21:06.379921 20599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:06.379998 20599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:06.416328 20599 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:45739
I20260807 11:21:06.416419 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:45739 every 8 connection(s)
I20260807 11:21:06.417686 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:06.422732 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.420882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:50769)
0807 11:21:06.421616 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.421851 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
W20260807 11:21:06.423338 20780 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:45069 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:45069: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
I20260807 11:21:06.424134 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.423829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:47327)
0807 11:21:06.423964 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.424028 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 11:21:06.425004 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.424671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:57113)
0807 11:21:06.424812 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.424892 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 11:21:06.425295 20780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:21:06.425357 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20599
I20260807 11:21:06.426034 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:42091
--local_ip_for_outbound_sockets=127.2.223.130
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=44819
--webserver_interface=127.2.223.130
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:06.782755 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:06.783067 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:06.783169 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:06.790544 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:06.790674 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:06.790791 20784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:06.800982 20784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:42091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=44819
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:06.802362 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:06.803851 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:06.818264 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:06.818300 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:06.818364 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:06.819622 20784 server_base.cc:1034] running on GCE node
I20260807 11:21:06.820343 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:06.821426 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:06.822666 20784 hybrid_clock.cc:648] HybridClock initialized: now 1786101666822606 us; error 55 us; skew 500 ppm
I20260807 11:21:06.823076 20784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:06 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101666, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:06.831234 20784 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:21:06.832967 20784 webserver.cc:460] Webserver started at http://127.2.223.130:44819/ using document root <none> and password file <none>
I20260807 11:21:06.833757 20784 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:06.833921 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:06.840075 20784 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 11:21:06.843195 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:06.844797 20784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:06.845028 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "d313f5cbb893410a97819e969c03974b"
format_stamp: "Formatted at 2026-08-07 11:20:20 on dist-test-slave-5l3k"
server_key: "e8fb548b269022ba45927168715256cf"
server_key_iv: "6aa61f5c2043c1ba2a70e6a009abf2d2"
server_key_version: "encryptionkey@0"
I20260807 11:21:06.846312 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:06.871791 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:06.876628 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:06.876956 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:06.878084 20784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:06.879638 20784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:06.879786 20784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:06.879942 20784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:06.880046 20784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:06.917902 20784 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:42091
I20260807 11:21:06.917951 20923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:42091 every 8 connection(s)
I20260807 11:21:06.919478 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:06.921499 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20784
I20260807 11:21:06.922359 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:33749
--local_ip_for_outbound_sockets=127.2.223.131
--tserver_master_addrs=127.2.223.190:45069
--webserver_port=36081
--webserver_interface=127.2.223.131
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:21:06.925282 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.923102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:57895)
0807 11:21:06.924049 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.924341 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":364,"threads_started":1}
W20260807 11:21:06.926054 20924 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:45069 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:45069: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
I20260807 11:21:06.927059 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.926713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:38855)
0807 11:21:06.926866 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.926939 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 11:21:06.927927 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:06.927624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:43721)
0807 11:21:06.927750 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:06.927823 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 11:21:06.928231 20924 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:21:07.292016 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:07.292371 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:07.292538 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:07.299831 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:07.299964 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:07.300086 20927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:07.310173 20927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:33749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=36081
--tserver_master_addrs=127.2.223.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:07.311487 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:07.312764 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:07.325994 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:07.326064 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:07.326023 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:07.326846 20927 server_base.cc:1034] running on GCE node
I20260807 11:21:07.327476 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:07.328472 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:07.329699 20927 hybrid_clock.cc:648] HybridClock initialized: now 1786101667329655 us; error 62 us; skew 500 ppm
I20260807 11:21:07.330006 20927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:07 dist-test-slave-5l3k krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101667, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:07.333220 20927 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:21:07.334708 20927 webserver.cc:460] Webserver started at http://127.2.223.131:36081/ using document root <none> and password file <none>
I20260807 11:21:07.335345 20927 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:07.335451 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:07.341231 20927 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 11:21:07.344004 20943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:07.345330 20927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:21:07.345469 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "c7c85cbd02d548d6a58d9589a07b4ff3"
format_stamp: "Formatted at 2026-08-07 11:20:21 on dist-test-slave-5l3k"
server_key: "52e59692f9002263e890951cfe3d7797"
server_key_iv: "2bf8462ae3a73c97a28d46789228ad3f"
server_key_version: "encryptionkey@0"
I20260807 11:21:07.346426 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:07.382509 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:07.386791 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:07.387027 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:07.388021 20927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:07.389453 20927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:07.389550 20927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:07.389638 20927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:07.389694 20927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:07.426926 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:07.426026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:52923)
0807 11:21:07.426612 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:07.426701 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":251,"threads_started":1}
I20260807 11:21:07.427987 20927 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:33749
I20260807 11:21:07.428109 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:33749 every 8 connection(s)
I20260807 11:21:07.429477 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:07.434229 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:07.432429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:51949)
0807 11:21:07.433147 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:07.433377 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
W20260807 11:21:07.434973 21067 heartbeater.cc:644] Failed to heartbeat to 127.2.223.190:45069 (0 consecutive failures): Network error: Failed to ping master at 127.2.223.190:45069: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
I20260807 11:21:07.435738 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:07.435426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:39435)
0807 11:21:07.435577 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:07.435642 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 11:21:07.436560 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:07.436235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:58017)
0807 11:21:07.436373 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:07.436453 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:21:07.436802 21067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:21:07.439917 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 20927
I20260807 11:21:07.929723 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:07.928826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:33429)
0807 11:21:07.929349 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:07.929449 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260807 11:21:08.428707 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:08.427939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:48073)
0807 11:21:08.428453 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:08.428536 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
I20260807 11:21:08.438320 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:08.437358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:39271)
0807 11:21:08.437901 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:08.438046 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260807 11:21:08.931521 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:08.930736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:41277)
0807 11:21:08.931242 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:08.931337 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260807 11:21:09.430570 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:09.429683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:54735)
0807 11:21:09.430260 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:09.430356 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":309,"threads_started":1}
I20260807 11:21:09.440209 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:09.439347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:46279)
0807 11:21:09.439917 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:09.440025 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":314,"threads_started":1}
I20260807 11:21:09.933104 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:09.932341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:35039)
0807 11:21:09.932803 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:09.932887 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260807 11:21:10.432351 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:10.431533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:40725)
0807 11:21:10.432101 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:10.432181 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
I20260807 11:21:10.441785 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:10.441061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:60961)
0807 11:21:10.441527 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:10.441610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":219,"threads_started":1}
I20260807 11:21:10.934872 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:10.934027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:40715)
0807 11:21:10.934572 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:10.934651 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260807 11:21:11.434121 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:11.433276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:34775)
0807 11:21:11.433867 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:11.433949 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":340,"threads_started":1}
I20260807 11:21:11.443423 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:11.442709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:49805)
0807 11:21:11.443168 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:11.443252 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260807 11:21:11.936486 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:11.935761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:33767)
0807 11:21:11.936230 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:11.936322 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260807 11:21:12.435678 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:12.434985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:51567)
0807 11:21:12.435416 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:12.435501 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260807 11:21:12.444939 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:12.444293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:57789)
0807 11:21:12.444707 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:12.444784 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260807 11:21:12.938405 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:12.937475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:57555)
0807 11:21:12.938098 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:12.938226 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":372,"threads_started":1}
I20260807 11:21:13.437670 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:13.436755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:40299)
0807 11:21:13.437328 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:13.437425 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260807 11:21:13.447000 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:13.446016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:56735)
0807 11:21:13.446689 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:13.446803 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260807 11:21:13.940451 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:13.939575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:41177)
0807 11:21:13.940122 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:13.940216 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":229,"threads_started":1}
I20260807 11:21:14.439965 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:14.439081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:52817)
0807 11:21:14.439646 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:14.439756 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":266,"threads_started":1}
I20260807 11:21:14.449056 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:14.448088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:48669)
0807 11:21:14.448727 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:14.448824 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":305,"threads_started":1}
I20260807 11:21:14.942260 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:14.941491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:37151)
0807 11:21:14.941983 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:14.942071 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":270,"threads_started":1}
I20260807 11:21:15.441717 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:15.440931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:38239)
0807 11:21:15.441458 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:15.441554 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260807 11:21:15.451046 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:15.450152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:35433)
0807 11:21:15.450772 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:15.450878 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":341,"threads_started":1}
I20260807 11:21:15.943974 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:15.943240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:56139)
0807 11:21:15.943709 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:15.943794 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":218,"threads_started":1}
I20260807 11:21:16.443409 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:16.442719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:48459)
0807 11:21:16.443136 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:16.443217 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260807 11:21:16.452682 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:16.451889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:50459)
0807 11:21:16.452383 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:16.452501 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":216,"threads_started":1}
I20260807 11:21:16.945834 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:16.945018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:41415)
0807 11:21:16.945573 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:16.945666 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
I20260807 11:21:17.445140 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:17.444366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:42811)
0807 11:21:17.444857 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:17.444946 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260807 11:21:17.454632 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:17.453713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:53519)
0807 11:21:17.454322 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:17.454412 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":372,"threads_started":1}
I20260807 11:21:17.947808 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:17.947008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:57671)
0807 11:21:17.947502 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:17.947615 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260807 11:21:18.447119 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:18.446225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:56669)
0807 11:21:18.446830 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:18.446925 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
I20260807 11:21:18.456676 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:18.455698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:49623)
0807 11:21:18.456358 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:18.456465 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":336,"threads_started":1}
I20260807 11:21:18.949818 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:18.949007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:59801)
0807 11:21:18.949520 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:18.949620 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260807 11:21:19.449039 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:19.448216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:45071)
0807 11:21:19.448716 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:19.448825 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":245,"threads_started":1}
I20260807 11:21:19.458776 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:19.457816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:36569)
0807 11:21:19.458382 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:19.458576 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":291,"threads_started":1}
I20260807 11:21:19.951908 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:19.951040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:39239)
0807 11:21:19.951603 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:19.951712 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260807 11:21:20.451010 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:20.450128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:36345)
0807 11:21:20.450691 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:20.450796 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":238,"threads_started":1}
I20260807 11:21:20.460695 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:20.459781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:52049)
0807 11:21:20.460383 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:20.460488 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":323,"threads_started":1}
I20260807 11:21:20.953589 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:20.952851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:57269)
0807 11:21:20.953313 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:20.953407 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260807 11:21:21.453428 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:21.452477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:47045)
0807 11:21:21.453055 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:21.453224 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":326,"threads_started":1}
I20260807 11:21:21.462754 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:21.461750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:33851)
0807 11:21:21.462396 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:21.462548 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":374,"threads_started":1}
I20260807 11:21:21.955852 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:21.955040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.130:45545)
0807 11:21:21.955543 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:21.955663 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260807 11:21:22.450630 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20599
I20260807 11:21:22.455538 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:22.454397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.129:37737)
0807 11:21:22.455051 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:22.455229 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":362,"threads_started":1}
I20260807 11:21:22.464895 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:22.464050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:45069 (local address 127.2.223.131:54757)
0807 11:21:22.464604 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:22.464698 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.190:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
I20260807 11:21:22.471534 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:22.451931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.2.223.129:45739)
0807 11:21:22.452690 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:22.452710 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:22.452763 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:21:22.454770 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:22.454781 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:22.455316 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:22.455725 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:22.457501 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.458940 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.460341 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.461104 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.461327 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:22.464407 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:22.464416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:22.464434 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:22.464481 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:22.467858 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.468631 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.468645 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.468653 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.468836 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.469255 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.469260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.469264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.469871 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:22.470208 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:22.470733 (+ 525us) server_negotiation.cc:299] Negotiation successful
0807 11:21:22.471209 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":378,"threads_started":1}
I20260807 11:21:22.473266 20775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:22.564114 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20599
I20260807 11:21:22.587242 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20784
I20260807 11:21:22.607074 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:22.588306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.2.223.130:42091)
0807 11:21:22.588975 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:22.588989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:22.589033 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:21:22.590843 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:22.590855 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:22.591332 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:22.591670 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:22.593199 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.594679 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.596111 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.596834 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.596982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:22.600076 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:22.600093 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:22.600112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:22.600157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:22.603311 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.604162 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.604169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.604177 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.604375 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.604862 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.604866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.604868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.605490 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:22.605776 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:22.606226 (+ 450us) server_negotiation.cc:299] Negotiation successful
0807 11:21:22.606765 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
I20260807 11:21:22.608649 20918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:22.696211 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20784
I20260807 11:21:22.719157 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 20927
I20260807 11:21:22.738288 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:22.720192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.2.223.131:33749)
0807 11:21:22.720821 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:22.720837 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:22.720881 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:21:22.722696 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:22.722707 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:22.723186 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:22.723491 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:22.724908 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.726131 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.727529 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:22.728233 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:22.728371 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:22.731597 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:22.731607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:22.731625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:22.731670 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:22.734993 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.735782 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.735790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.735797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.736036 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:22.736412 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:22.736416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:22.736418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:22.737002 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:22.737250 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:22.737591 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 11:21:22.737955 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":320,"threads_started":1}
I20260807 11:21:22.739991 21060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:22.824617 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 20927
2026-08-07T11:21:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65150 ms)
[----------] 30 tests from SecurityITest (313119 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:22 dist-test-slave-5l3k krb5kdc[21126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:22 dist-test-slave-5l3k krb5kdc[21126](info): set up 2 sockets
Aug 07 11:21:22 dist-test-slave-5l3k krb5kdc[21126](info): commencing operation
krb5kdc: starting...
W20260807 11:21:24.021468 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:24 dist-test-slave-5l3k krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:21:24.402415 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46501
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:24.770840 21139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:24.771176 21139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:24.771271 21139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:24.778975 21139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:24.779109 21139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:24.779161 21139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:24.779227 21139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:24.779275 21139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:24.789791 21139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:24.791290 21139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:24.792680 21139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:24.805634 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:24.805742 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:24.805770 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:24.806799 21139 server_base.cc:1034] running on GCE node
I20260807 11:21:24.807715 21139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:24.809455 21139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:24.810703 21139 hybrid_clock.cc:648] HybridClock initialized: now 1786101684810635 us; error 59 us; skew 500 ppm
I20260807 11:21:24.811082 21139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:24 dist-test-slave-5l3k krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:24.814574 21139 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:21:24.816277 21139 webserver.cc:460] Webserver started at http://127.2.223.190:41117/ using document root <none> and password file <none>
I20260807 11:21:24.817071 21139 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:24.817234 21139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:24.817582 21139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:24.820989 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "447e0b04823f45999109ba9f3ca1c8cc"
format_stamp: "Formatted at 2026-08-07 11:21:24 on dist-test-slave-5l3k"
server_key: "f758055aa141d9a671c2150abffc2ef4"
server_key_iv: "b85323ade3cf8fbe30169960c3bafa29"
server_key_version: "encryptionkey@0"
I20260807 11:21:24.821897 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "447e0b04823f45999109ba9f3ca1c8cc"
format_stamp: "Formatted at 2026-08-07 11:21:24 on dist-test-slave-5l3k"
server_key: "f758055aa141d9a671c2150abffc2ef4"
server_key_iv: "b85323ade3cf8fbe30169960c3bafa29"
server_key_version: "encryptionkey@0"
I20260807 11:21:24.827829 21139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:21:24.831875 21155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:24.833482 21139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:21:24.833725 21139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "447e0b04823f45999109ba9f3ca1c8cc"
format_stamp: "Formatted at 2026-08-07 11:21:24 on dist-test-slave-5l3k"
server_key: "f758055aa141d9a671c2150abffc2ef4"
server_key_iv: "b85323ade3cf8fbe30169960c3bafa29"
server_key_version: "encryptionkey@0"
I20260807 11:21:24.833966 21139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:24.853570 21139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:24.860136 21139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:24.860505 21139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:24.881915 21139 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:46501
I20260807 11:21:24.882002 21211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:46501 every 8 connection(s)
I20260807 11:21:24.883530 21139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:24.887776 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:24.890044 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21139
I20260807 11:21:24.890295 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:24.891084 2942 external_mini_cluster.cc:1428] Setting key dd722f708b6bf38c5be83f2095d604de
I20260807 11:21:24.901023 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: Bootstrap starting.
I20260807 11:21:24.905687 21212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:24.907299 21212 log.cc:824] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:21:24 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:24.911068 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: No bootstrap required, opened a new log
I20260807 11:21:24.918465 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:24.893314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.2.223.190:46501)
0807 11:21:24.894228 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:24.894242 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:24.894286 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:21:24.896160 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:24.896186 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:24.896671 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:24.897069 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:24.899217 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:24.900648 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:24.902736 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:24.903550 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:24.903714 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:24.909817 (+ 6103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:24.909828 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:24.909847 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:24.909889 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:24.913357 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:24.914541 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:24.914559 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:24.914567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:24.914771 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:24.915270 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:24.915274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:24.915277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:24.915904 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:24.916201 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:24.916740 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 11:21:24.917234 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":433,"threads_started":1}
I20260807 11:21:24.921064 21212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } }
I20260807 11:21:24.921490 21212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:24.921640 21212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447e0b04823f45999109ba9f3ca1c8cc, State: Initialized, Role: FOLLOWER
I20260807 11:21:24.922286 21212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } }
I20260807 11:21:24.922588 21212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:24.922705 21212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:24.922842 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:24.925127 21212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } }
I20260807 11:21:24.925681 21212 leader_election.cc:302] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 447e0b04823f45999109ba9f3ca1c8cc; no voters:
I20260807 11:21:24.926359 21212 leader_election.cc:288] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:24.926602 21217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:24.927927 21217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [term 1 LEADER]: Becoming Leader. State: Replica: 447e0b04823f45999109ba9f3ca1c8cc, State: Running, Role: LEADER
I20260807 11:21:24.928515 21217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } }
I20260807 11:21:24.928967 21212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:24.931360 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 447e0b04823f45999109ba9f3ca1c8cc. Latest consensus state: current_term: 1 leader_uuid: "447e0b04823f45999109ba9f3ca1c8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } } }
I20260807 11:21:24.932130 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:24.932229 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "447e0b04823f45999109ba9f3ca1c8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447e0b04823f45999109ba9f3ca1c8cc" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46501 } } }
I20260807 11:21:24.932605 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:24.935971 21227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:24.940917 21227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:24.950249 21227 catalog_manager.cc:1347] Generated new cluster ID: 1c3774f6f8d14abf918ed432ee54afc8
I20260807 11:21:24.950392 21227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:24.969774 21227 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:24.971383 21227 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:24.980747 21227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: Generated new TSK 0
I20260807 11:21:24.981747 21227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:21:25.086618 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:25.495972 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:25.496330 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:25.496435 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:25.504418 21240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:25.504575 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:25.504698 21240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:25.515581 21240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:25.517108 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:25.518589 21240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:25.533802 21249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:25.533907 21246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:25.533907 21247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:25.535219 21240 server_base.cc:1034] running on GCE node
I20260807 11:21:25.536003 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:25.537225 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:25.538594 21240 hybrid_clock.cc:648] HybridClock initialized: now 1786101685538519 us; error 61 us; skew 500 ppm
I20260807 11:21:25.538995 21240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:25 dist-test-slave-5l3k krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:25.542800 21240 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:21:25.544706 21240 webserver.cc:460] Webserver started at http://127.2.223.129:35183/ using document root <none> and password file <none>
I20260807 11:21:25.545559 21240 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:25.545699 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:25.546011 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:25.549458 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "2f3acdff6f72456482098b21da2ad9a3"
format_stamp: "Formatted at 2026-08-07 11:21:25 on dist-test-slave-5l3k"
server_key: "bada163c68e26a51cd600d070a8b160b"
server_key_iv: "d0ac49b830e9f727aafb5846f5f8691b"
server_key_version: "encryptionkey@0"
I20260807 11:21:25.550422 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "2f3acdff6f72456482098b21da2ad9a3"
format_stamp: "Formatted at 2026-08-07 11:21:25 on dist-test-slave-5l3k"
server_key: "bada163c68e26a51cd600d070a8b160b"
server_key_iv: "d0ac49b830e9f727aafb5846f5f8691b"
server_key_version: "encryptionkey@0"
I20260807 11:21:25.556928 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:25.561539 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:25.563400 21240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260807 11:21:25.563700 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "2f3acdff6f72456482098b21da2ad9a3"
format_stamp: "Formatted at 2026-08-07 11:21:25 on dist-test-slave-5l3k"
server_key: "bada163c68e26a51cd600d070a8b160b"
server_key_iv: "d0ac49b830e9f727aafb5846f5f8691b"
server_key_version: "encryptionkey@0"
I20260807 11:21:25.563973 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:25.596488 21240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:25.602008 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:25.602409 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:25.603609 21240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:25.605299 21240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:25.605433 21240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:25.605610 21240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:25.605700 21240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:25.644618 21240 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:43109
I20260807 11:21:25.644680 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:43109 every 8 connection(s)
I20260807 11:21:25.646193 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:25.649225 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21240
I20260807 11:21:25.649472 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:25.650336 2942 external_mini_cluster.cc:1428] Setting key 90f03c1642c8407be74a272d20a13c21
Aug 07 11:21:25 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:25.673903 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:25.649281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:60455 (local address 127.2.223.190:46501)
0807 11:21:25.649870 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:25.649876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:25.651301 (+ 1425us) server_negotiation.cc:407] Connection header received
0807 11:21:25.653386 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:25.653394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:25.653513 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:25.653642 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:25.655613 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:25.656809 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:25.659163 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:25.659947 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:25.660137 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:25.664918 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:25.664929 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:25.664936 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:25.664997 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:25.669167 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:25.670442 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:25.670452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:25.670456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:25.670736 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:25.671434 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:25.671443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:25.671449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:25.671925 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:25.672105 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:25.673036 (+ 931us) server_negotiation.cc:299] Negotiation successful
0807 11:21:25.673357 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":275,"threads_started":1}
I20260807 11:21:25.674765 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:25.650016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46501 (local address 127.2.223.129:60455)
0807 11:21:25.651036 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:25.651100 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:25.653039 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:25.653924 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:25.653934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:25.654671 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:25.655361 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:25.655383 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:25.657107 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:25.657112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:25.658896 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:25.658919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:25.660302 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:25.660308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:25.660586 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:25.661544 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:25.661576 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:25.664502 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:25.669477 (+ 4975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:25.669486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:25.669509 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:25.670186 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:25.670945 (+ 759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:25.670949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:25.670952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:25.671258 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:25.672101 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:25.672109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:25.672534 (+ 425us) client_negotiation.cc:769] Sending connection context
0807 11:21:25.672816 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 11:21:25.673333 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":417,"threads_started":1}
I20260807 11:21:25.676868 21380 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46501
I20260807 11:21:25.677274 21380 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:25.678206 21380 heartbeater.cc:511] Master 127.2.223.190:46501 requested a full tablet report, sending...
I20260807 11:21:25.680946 21174 ts_manager.cc:194] Registered new tserver with Master: 2f3acdff6f72456482098b21da2ad9a3 (127.2.223.129:43109)
I20260807 11:21:25.682703 21174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:60455
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:21:25.739336 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:26.133256 21386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:26.133623 21386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:26.133821 21386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:26.142181 21386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:26.142333 21386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:26.142593 21386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:26.153610 21386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:26.155301 21386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:26.156937 21386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:26.173195 21391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:26.173110 21394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:26.173173 21392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:26.174748 21386 server_base.cc:1034] running on GCE node
I20260807 11:21:26.175545 21386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:26.176764 21386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:26.178040 21386 hybrid_clock.cc:648] HybridClock initialized: now 1786101686177989 us; error 67 us; skew 500 ppm
I20260807 11:21:26.178440 21386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:26 dist-test-slave-5l3k krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:26.182355 21386 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:21:26.184373 21386 webserver.cc:460] Webserver started at http://127.2.223.130:42533/ using document root <none> and password file <none>
I20260807 11:21:26.185246 21386 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:26.185429 21386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:26.185819 21386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:26.189424 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "8b3db7b70ca545268fce4adffe455990"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "660c75aa58014131b2f343a4f5e0951e"
server_key_iv: "afae73994f42a2ac4ec34d2456b1ed84"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.190465 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "8b3db7b70ca545268fce4adffe455990"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "660c75aa58014131b2f343a4f5e0951e"
server_key_iv: "afae73994f42a2ac4ec34d2456b1ed84"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.197072 21386 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:26.201733 21401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.203611 21386 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260807 11:21:26.203981 21386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "8b3db7b70ca545268fce4adffe455990"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "660c75aa58014131b2f343a4f5e0951e"
server_key_iv: "afae73994f42a2ac4ec34d2456b1ed84"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.204283 21386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:26.229633 21386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:26.235169 21386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:26.235531 21386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:26.236754 21386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:26.238721 21386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:26.238860 21386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.239032 21386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:26.239111 21386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.283396 21386 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:35743
I20260807 11:21:26.285635 21386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:26.283396 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:35743 every 8 connection(s)
I20260807 11:21:26.290438 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21386
I20260807 11:21:26.290688 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:26.291323 2942 external_mini_cluster.cc:1428] Setting key 4c265f80722b6b1b98d9698edfcabf34
Aug 07 11:21:26 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:26.311004 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:26.289075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:35123 (local address 127.2.223.190:46501)
0807 11:21:26.289699 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:26.289708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:26.291220 (+ 1512us) server_negotiation.cc:407] Connection header received
0807 11:21:26.293846 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:26.293853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:26.293942 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:26.294089 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:26.295975 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.296781 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.298734 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.299252 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.299419 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:26.303756 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:26.303763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:26.303767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:26.303813 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:26.306865 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.307885 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.307890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.307893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.308047 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.308675 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.308679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.308682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.309073 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:26.309273 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:26.310225 (+ 952us) server_negotiation.cc:299] Negotiation successful
0807 11:21:26.310556 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":295,"threads_started":1}
I20260807 11:21:26.312340 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:26.289727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46501 (local address 127.2.223.130:35123)
0807 11:21:26.290700 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:26.290765 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:26.293300 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:26.294391 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:26.294401 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:26.295088 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:26.295754 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:26.295775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.297003 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.297008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:26.298448 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:26.298465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.299552 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.299561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:26.299847 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:26.300721 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:26.300750 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:26.303376 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:26.307098 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:26.307111 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:26.307146 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:26.307705 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:26.308191 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:26.308196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:26.308200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:26.308511 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:26.309234 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:26.309244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:26.309760 (+ 516us) client_negotiation.cc:769] Sending connection context
0807 11:21:26.310144 (+ 384us) client_negotiation.cc:240] Negotiation successful
0807 11:21:26.310866 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":407,"threads_started":1}
I20260807 11:21:26.314409 21521 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46501
I20260807 11:21:26.314896 21521 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:26.316058 21521 heartbeater.cc:511] Master 127.2.223.190:46501 requested a full tablet report, sending...
I20260807 11:21:26.318523 21174 ts_manager.cc:194] Registered new tserver with Master: 8b3db7b70ca545268fce4adffe455990 (127.2.223.130:35743)
I20260807 11:21:26.319957 21174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:35123
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:21:26.394753 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:21:26.686789 21380 heartbeater.cc:503] Master 127.2.223.190:46501 was elected leader, sending a full tablet report...
W20260807 11:21:26.793437 21529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:26.793777 21529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:26.793952 21529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:26.801844 21529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:26.801998 21529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:26.802208 21529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:26.813356 21529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:26.815500 21529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:26.817183 21529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:26.833712 21535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:26.833621 21538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:26.833684 21534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:26.835276 21529 server_base.cc:1034] running on GCE node
I20260807 11:21:26.836045 21529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:26.837189 21529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:26.838613 21529 hybrid_clock.cc:648] HybridClock initialized: now 1786101686838417 us; error 75 us; skew 500 ppm
I20260807 11:21:26.839035 21529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:26 dist-test-slave-5l3k krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:26.847117 21529 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:21:26.849279 21529 webserver.cc:460] Webserver started at http://127.2.223.131:34551/ using document root <none> and password file <none>
I20260807 11:21:26.850200 21529 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:26.850397 21529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:26.850806 21529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:26.854842 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "61f7b4e4a89d4800ba189929d3fc14ae"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "40a0cc8a653532df7ce0fc7c4709460e"
server_key_iv: "64c8a5d3129e391734ac4d589716426e"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.856227 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "61f7b4e4a89d4800ba189929d3fc14ae"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "40a0cc8a653532df7ce0fc7c4709460e"
server_key_iv: "64c8a5d3129e391734ac4d589716426e"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.863477 21529 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260807 11:21:26.867967 21545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.869916 21529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260807 11:21:26.870110 21529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "61f7b4e4a89d4800ba189929d3fc14ae"
format_stamp: "Formatted at 2026-08-07 11:21:26 on dist-test-slave-5l3k"
server_key: "40a0cc8a653532df7ce0fc7c4709460e"
server_key_iv: "64c8a5d3129e391734ac4d589716426e"
server_key_version: "encryptionkey@0"
I20260807 11:21:26.870332 21529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:26.885331 21529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:26.890390 21529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:26.890818 21529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:26.891956 21529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:26.893574 21529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:26.893661 21529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.893806 21529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:26.893877 21529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:26.933669 21529 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:43297
I20260807 11:21:26.933734 21661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:43297 every 8 connection(s)
I20260807 11:21:26.935241 21529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:26.936606 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21529
I20260807 11:21:26.936830 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:26.937423 2942 external_mini_cluster.cc:1428] Setting key 6a8ae6a04f1f18f556cad6566d236c24
Aug 07 11:21:26 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:26.959812 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:26.938598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:55879 (local address 127.2.223.190:46501)
0807 11:21:26.939208 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:26.939213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:26.940510 (+ 1297us) server_negotiation.cc:407] Connection header received
0807 11:21:26.942814 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:26.942820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:26.942988 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:26.943166 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:26.945064 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.945908 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.947875 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.948333 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.948490 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:26.952855 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:26.952863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:26.952870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:26.952921 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:26.956179 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.957186 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.957190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.957192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.957356 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.957848 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.957851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.957853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.958149 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:26.958305 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:26.959298 (+ 993us) server_negotiation.cc:299] Negotiation successful
0807 11:21:26.959502 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260807 11:21:26.960961 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:26.939438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46501 (local address 127.2.223.131:55879)
0807 11:21:26.940343 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:26.940389 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:26.942389 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:26.943502 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:26.943512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:26.944153 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:26.944833 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:26.944856 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.946127 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.946132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:26.947640 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:26.947660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.948653 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.948662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:26.948938 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:26.949761 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:26.949788 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:26.952604 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:26.956402 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:26.956412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:26.956429 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:26.957019 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:26.957471 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:26.957474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:26.957477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:26.957716 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:26.958294 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:26.958301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:26.958770 (+ 469us) client_negotiation.cc:769] Sending connection context
0807 11:21:26.959175 (+ 405us) client_negotiation.cc:240] Negotiation successful
0807 11:21:26.959695 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":400,"threads_started":1}
I20260807 11:21:26.962862 21662 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46501
I20260807 11:21:26.963322 21662 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:26.964382 21662 heartbeater.cc:511] Master 127.2.223.190:46501 requested a full tablet report, sending...
I20260807 11:21:26.966681 21174 ts_manager.cc:194] Registered new tserver with Master: 61f7b4e4a89d4800ba189929d3fc14ae (127.2.223.131:43297)
I20260807 11:21:26.967845 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:26.967826 21174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:55879
I20260807 11:21:26.988473 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:26.973316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.2.223.190:46501)
0807 11:21:26.973505 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:26.973509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:26.974026 (+ 517us) server_negotiation.cc:407] Connection header received
0807 11:21:26.974197 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:26.974226 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:26.974356 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:26.974521 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:26.976305 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.977066 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.978651 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:26.979096 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:26.979230 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:26.982091 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:26.982097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:26.982101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:26.982145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:26.985279 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.986190 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.986194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.986197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.986373 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:26.986888 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:26.986891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:26.986893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:26.987295 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:26.987504 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:26.987987 (+ 483us) server_negotiation.cc:299] Negotiation successful
0807 11:21:26.988195 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:21:26.993793 21174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:27.001950 21174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:27.039929 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.018072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.2.223.129:43109)
0807 11:21:27.027682 (+ 9610us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.027691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.027728 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:21:27.027863 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.027875 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.028190 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.029650 (+ 1460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:27.029971 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.032346 (+ 2375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.036699 (+ 4353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.038624 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.038791 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.039196 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.039334 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.039522 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9403,"thread_start_us":9185,"threads_started":1}
I20260807 11:21:27.039996 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.017504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:60062)
0807 11:21:27.019431 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.019505 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:27.019765 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:27.028494 (+ 8729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:27.028507 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:27.028573 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:27.029265 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.029293 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.032607 (+ 3314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.032618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.036344 (+ 3726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.036358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.038845 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.038852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.039072 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.039127 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 11:21:27.039289 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 11:21:27.039513 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":10048,"threads_started":1}
I20260807 11:21:27.041630 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.019106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.2.223.131:43297)
0807 11:21:27.027335 (+ 8229us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.027345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.027981 (+ 636us) server_negotiation.cc:407] Connection header received
0807 11:21:27.028302 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.028311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.028545 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.028783 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:27.032144 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.034730 (+ 2586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.038139 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.039765 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.039967 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.040228 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.041091 (+ 863us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.041282 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7974,"thread_start_us":7685,"threads_started":1}
I20260807 11:21:27.041615 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.018359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:43297 (local address 127.0.0.1:54496)
0807 11:21:27.027823 (+ 9464us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.027842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:27.028046 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:27.029262 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:27.029270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:27.029310 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:27.031769 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.031799 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.035039 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.035048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.037921 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.037941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.040716 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.040720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.040864 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.040889 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:21:27.041022 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 11:21:27.041179 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9341,"thread_start_us":963,"threads_started":1}
I20260807 11:21:27.044937 21309 tablet_service.cc:1467] Processing CreateTablet for tablet d1ab0446121947258a125a0106213ce3 (DEFAULT_TABLE table=test-table [id=6b7e5273bec94a41ae35c1bf8f666f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:27.045768 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.028286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35743 (local address 127.0.0.1:37560)
0807 11:21:27.035155 (+ 6869us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.035184 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:27.035958 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:27.037491 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:27.037496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:27.037524 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:27.038011 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.038024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.040028 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.040036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.042731 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.042750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.044882 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.044889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.045094 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.045131 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:21:27.045251 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:21:27.045407 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6671,"thread_start_us":7015,"threads_started":1}
I20260807 11:21:27.045852 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.028339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.2.223.130:35743)
0807 11:21:27.036720 (+ 8381us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.036733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.036772 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:21:27.036892 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.036902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.037212 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.037494 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:27.038210 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.039835 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.042955 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.044609 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.044770 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.044983 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.045324 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.045502 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7324,"thread_start_us":7156,"threads_started":1}
I20260807 11:21:27.046698 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ab0446121947258a125a0106213ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:27.046182 21594 tablet_service.cc:1467] Processing CreateTablet for tablet d1ab0446121947258a125a0106213ce3 (DEFAULT_TABLE table=test-table [id=6b7e5273bec94a41ae35c1bf8f666f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:27.047684 21594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ab0446121947258a125a0106213ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:27.051246 21453 tablet_service.cc:1467] Processing CreateTablet for tablet d1ab0446121947258a125a0106213ce3 (DEFAULT_TABLE table=test-table [id=6b7e5273bec94a41ae35c1bf8f666f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:27.052955 21453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ab0446121947258a125a0106213ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:27.061376 21682 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Bootstrap starting.
I20260807 11:21:27.065049 21682 tablet_bootstrap.cc:654] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:27.066164 21683 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Bootstrap starting.
I20260807 11:21:27.066712 21682 log.cc:824] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:27.070405 21683 tablet_bootstrap.cc:654] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:27.070667 21682 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: No bootstrap required, opened a new log
I20260807 11:21:27.071081 21682 ts_tablet_manager.cc:1397] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260807 11:21:27.072050 21683 log.cc:824] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:27.072392 21686 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Bootstrap starting.
I20260807 11:21:27.075863 21683 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: No bootstrap required, opened a new log
I20260807 11:21:27.076186 21683 ts_tablet_manager.cc:1397] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 11:21:27.076289 21686 tablet_bootstrap.cc:654] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:27.077721 21686 log.cc:824] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:27.080402 21682 raft_consensus.cc:348] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.080839 21682 raft_consensus.cc:374] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:27.080942 21682 raft_consensus.cc:729] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3acdff6f72456482098b21da2ad9a3, State: Initialized, Role: FOLLOWER
I20260807 11:21:27.081596 21686 tablet_bootstrap.cc:492] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: No bootstrap required, opened a new log
I20260807 11:21:27.081619 21682 consensus_queue.cc:260] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.081907 21686 ts_tablet_manager.cc:1397] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260807 11:21:27.085142 21682 ts_tablet_manager.cc:1428] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260807 11:21:27.088799 21683 raft_consensus.cc:348] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.089264 21683 raft_consensus.cc:374] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:27.089407 21683 raft_consensus.cc:729] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f7b4e4a89d4800ba189929d3fc14ae, State: Initialized, Role: FOLLOWER
I20260807 11:21:27.090157 21683 consensus_queue.cc:260] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.091477 21662 heartbeater.cc:503] Master 127.2.223.190:46501 was elected leader, sending a full tablet report...
I20260807 11:21:27.092206 21683 ts_tablet_manager.cc:1428] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260807 11:21:27.092335 21686 raft_consensus.cc:348] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.092777 21686 raft_consensus.cc:374] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:27.092900 21686 raft_consensus.cc:729] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3db7b70ca545268fce4adffe455990, State: Initialized, Role: FOLLOWER
I20260807 11:21:27.093575 21686 consensus_queue.cc:260] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.094815 21521 heartbeater.cc:503] Master 127.2.223.190:46501 was elected leader, sending a full tablet report...
I20260807 11:21:27.095602 21686 ts_tablet_manager.cc:1428] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260807 11:21:27.149422 21381 tablet.cc:2369] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:27.187116 21663 tablet.cc:2369] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:27.288395 21522 tablet.cc:2369] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:27.323289 21689 raft_consensus.cc:482] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:27.323659 21689 raft_consensus.cc:504] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.325651 21689 leader_election.cc:288] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b3db7b70ca545268fce4adffe455990 (127.2.223.130:35743), 61f7b4e4a89d4800ba189929d3fc14ae (127.2.223.131:43297)
I20260807 11:21:27.333748 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.326301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:45815 (local address 127.2.223.130:35743)
0807 11:21:27.326579 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.326588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.327539 (+ 951us) server_negotiation.cc:407] Connection header received
0807 11:21:27.327787 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.327791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.327854 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.327963 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:27.328633 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.329518 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.331719 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.332668 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.332872 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.332957 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.333321 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.333463 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:21:27.333889 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.326320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:43163 (local address 127.2.223.131:43297)
0807 11:21:27.326604 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.326613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.327583 (+ 970us) server_negotiation.cc:407] Connection header received
0807 11:21:27.327850 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.327855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.327930 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.328033 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:27.328738 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.329613 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.331904 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.332829 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.333033 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.333115 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.333460 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.333614 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:21:27.333940 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.326256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35743 (local address 127.2.223.129:45815)
0807 11:21:27.327403 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.327425 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:27.327632 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:27.327981 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:27.327985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:27.328010 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:27.328460 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.328471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.329658 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.329664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.331519 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.331537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.332897 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.332903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.333143 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.333170 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 11:21:27.333287 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:21:27.333440 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":1,"thread_start_us":815,"threads_started":1}
I20260807 11:21:27.334142 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.326298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:43297 (local address 127.2.223.129:43163)
0807 11:21:27.327461 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.327477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:27.327701 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:27.328047 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:27.328051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:27.328080 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:27.328541 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.328551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.329764 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.329772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.331696 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:27.331711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.333070 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.333077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:27.333296 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.333322 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 11:21:27.333428 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 11:21:27.333591 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":19,"spinlock_wait_cycles":48512,"thread_start_us":601,"threads_started":1}
I20260807 11:21:27.335456 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ab0446121947258a125a0106213ce3" candidate_uuid: "2f3acdff6f72456482098b21da2ad9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3db7b70ca545268fce4adffe455990" is_pre_election: true
I20260807 11:21:27.335458 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ab0446121947258a125a0106213ce3" candidate_uuid: "2f3acdff6f72456482098b21da2ad9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" is_pre_election: true
I20260807 11:21:27.336123 21616 raft_consensus.cc:2454] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3acdff6f72456482098b21da2ad9a3 in term 0.
I20260807 11:21:27.336123 21475 raft_consensus.cc:2454] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3acdff6f72456482098b21da2ad9a3 in term 0.
I20260807 11:21:27.337064 21259 leader_election.cc:302] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f3acdff6f72456482098b21da2ad9a3, 61f7b4e4a89d4800ba189929d3fc14ae, 8b3db7b70ca545268fce4adffe455990; no voters:
I20260807 11:21:27.337663 21689 raft_consensus.cc:2781] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:27.337867 21689 raft_consensus.cc:482] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:27.337987 21689 raft_consensus.cc:3037] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:27.340521 21689 raft_consensus.cc:504] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.341231 21689 leader_election.cc:288] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [CANDIDATE]: Term 1 election: Requested vote from peers 8b3db7b70ca545268fce4adffe455990 (127.2.223.130:35743), 61f7b4e4a89d4800ba189929d3fc14ae (127.2.223.131:43297)
I20260807 11:21:27.341881 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ab0446121947258a125a0106213ce3" candidate_uuid: "2f3acdff6f72456482098b21da2ad9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f7b4e4a89d4800ba189929d3fc14ae"
I20260807 11:21:27.341884 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ab0446121947258a125a0106213ce3" candidate_uuid: "2f3acdff6f72456482098b21da2ad9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3db7b70ca545268fce4adffe455990"
I20260807 11:21:27.342121 21616 raft_consensus.cc:3037] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:27.342134 21475 raft_consensus.cc:3037] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:27.344729 21616 raft_consensus.cc:2454] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3acdff6f72456482098b21da2ad9a3 in term 1.
I20260807 11:21:27.344722 21475 raft_consensus.cc:2454] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3acdff6f72456482098b21da2ad9a3 in term 1.
I20260807 11:21:27.345284 21259 leader_election.cc:302] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3acdff6f72456482098b21da2ad9a3, 8b3db7b70ca545268fce4adffe455990; no voters:
I20260807 11:21:27.345667 21689 raft_consensus.cc:2781] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:27.346242 21689 raft_consensus.cc:686] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3acdff6f72456482098b21da2ad9a3, State: Running, Role: LEADER
I20260807 11:21:27.346868 21689 consensus_queue.cc:237] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } }
I20260807 11:21:27.351974 21173 catalog_manager.cc:5495] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3acdff6f72456482098b21da2ad9a3 (127.2.223.129). New cstate: current_term: 1 leader_uuid: "2f3acdff6f72456482098b21da2ad9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3acdff6f72456482098b21da2ad9a3" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 43109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 } health_report { overall_health: UNKNOWN } } }
I20260807 11:21:27.408836 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.401895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.2.223.129:43109)
0807 11:21:27.402188 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.402195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.402269 (+ 74us) server_negotiation.cc:407] Connection header received
0807 11:21:27.402644 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.402649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.402761 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.402964 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:27.403422 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.404213 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.406103 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.406642 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.406819 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.407483 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:27.408059 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:27.408310 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.408366 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.408466 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260807 11:21:27.418555 21616 raft_consensus.cc:1261] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 1 FOLLOWER]: Refusing update from remote peer 2f3acdff6f72456482098b21da2ad9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:27.418552 21475 raft_consensus.cc:1261] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 1 FOLLOWER]: Refusing update from remote peer 2f3acdff6f72456482098b21da2ad9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:27.419746 21694 consensus_queue.cc:1035] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3db7b70ca545268fce4adffe455990" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:27.420262 21689 consensus_queue.cc:1035] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f7b4e4a89d4800ba189929d3fc14ae" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:27.450399 21173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:27.450817 21173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:27.455446 21173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447e0b04823f45999109ba9f3ca1c8cc: Sending DeleteTablet for 3 replicas of tablet d1ab0446121947258a125a0106213ce3
I20260807 11:21:27.457010 21594 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ab0446121947258a125a0106213ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:27 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:54496
I20260807 11:21:27.457147 21453 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ab0446121947258a125a0106213ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:27 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37560
I20260807 11:21:27.457378 21309 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ab0446121947258a125a0106213ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:27 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:60062
I20260807 11:21:27.462927 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21240
I20260807 11:21:27.470002 21709 tablet_replica.cc:330] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: stopping tablet replica
I20260807 11:21:27.470635 21709 raft_consensus.cc:2229] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:27.471311 21709 raft_consensus.cc:2258] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:27.472256 21710 tablet_replica.cc:330] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: stopping tablet replica
I20260807 11:21:27.472932 21710 raft_consensus.cc:2229] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:27.474077 21710 raft_consensus.cc:2258] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:27.475322 21708 tablet_replica.cc:330] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: stopping tablet replica
I20260807 11:21:27.475840 21708 raft_consensus.cc:2229] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:27.476426 21708 raft_consensus.cc:2258] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:27.477304 21709 ts_tablet_manager.cc:1905] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:27.477340 21710 ts_tablet_manager.cc:1905] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:27.479001 21708 ts_tablet_manager.cc:1905] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:21:27 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:21:27.485554 21709 ts_tablet_manager.cc:1918] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:27.485587 21710 ts_tablet_manager.cc:1918] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:27.485797 21710 log.cc:1196] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/d1ab0446121947258a125a0106213ce3
I20260807 11:21:27.486011 21709 log.cc:1196] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/d1ab0446121947258a125a0106213ce3
I20260807 11:21:27.486776 21709 ts_tablet_manager.cc:1939] T d1ab0446121947258a125a0106213ce3 P 8b3db7b70ca545268fce4adffe455990: Deleting consensus metadata
I20260807 11:21:27.486958 21708 ts_tablet_manager.cc:1918] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:27.487241 21708 log.cc:1196] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/d1ab0446121947258a125a0106213ce3
I20260807 11:21:27.487874 21708 ts_tablet_manager.cc:1939] T d1ab0446121947258a125a0106213ce3 P 61f7b4e4a89d4800ba189929d3fc14ae: Deleting consensus metadata
I20260807 11:21:27.490105 21157 catalog_manager.cc:4841] TS 61f7b4e4a89d4800ba189929d3fc14ae (127.2.223.131:43297): tablet d1ab0446121947258a125a0106213ce3 (table test-table [id=6b7e5273bec94a41ae35c1bf8f666f9e]) successfully deleted
I20260807 11:21:27.492108 21158 catalog_manager.cc:4841] TS 8b3db7b70ca545268fce4adffe455990 (127.2.223.130:35743): tablet d1ab0446121947258a125a0106213ce3 (table test-table [id=6b7e5273bec94a41ae35c1bf8f666f9e]) successfully deleted
I20260807 11:21:27.486430 21710 ts_tablet_manager.cc:1939] T d1ab0446121947258a125a0106213ce3 P 2f3acdff6f72456482098b21da2ad9a3: Deleting consensus metadata
I20260807 11:21:27.494239 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.464713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.2.223.129:43109)
0807 11:21:27.464944 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.464954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.470314 (+ 5360us) server_negotiation.cc:407] Connection header received
0807 11:21:27.470771 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.470780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.470876 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.470974 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:27.473692 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.475335 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.477552 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.478370 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.478525 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.484780 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:27.484800 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:27.484822 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:27.484883 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:27.489655 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.491006 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.491026 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.491031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.491295 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.491879 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.491884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.491887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.492460 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:27.492806 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.493518 (+ 712us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.493807 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":54}
I20260807 11:21:27.495525 21373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:27.601775 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21240
W20260807 11:21:27.625748 21158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.223.129:43109 (error 108)
I20260807 11:21:27.626739 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21386
I20260807 11:21:27.628249 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.627307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42668)
0807 11:21:27.627872 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.628022 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":709,"threads_started":1}
W20260807 11:21:27.628556 21158 catalog_manager.cc:4584] TS 2f3acdff6f72456482098b21da2ad9a3 (127.2.223.129:43109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1ab0446121947258a125a0106213ce3: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Aug 07 11:21:27 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:21:27.643987 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.627647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.2.223.130:35743)
0807 11:21:27.627816 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.627820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.627841 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:21:27.628055 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.628059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.628127 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.628250 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:27.629783 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.630570 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.631949 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.632431 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.632547 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.637279 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:27.637299 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:27.637317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:27.637365 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:27.640649 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.641657 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.641665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.641668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.641845 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.642319 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.642324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.642326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.642929 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:27.643206 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.643508 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.643706 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 11:21:27.644821 21516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:27.682983 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.682363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42684)
0807 11:21:27.682652 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.682786 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 11:21:27.684360 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.683867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42690)
0807 11:21:27.684047 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.684166 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 11:21:27.745440 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.744843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42692)
0807 11:21:27.745097 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.745221 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 11:21:27.745456 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21386
I20260807 11:21:27.755988 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.755522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42700)
0807 11:21:27.755711 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.755800 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 11:21:27.768420 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21529
Aug 07 11:21:27 dist-test-slave-5l3k krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:21:27.785859 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.769335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.2.223.131:43297)
0807 11:21:27.769547 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:27.769555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:27.769579 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:21:27.769815 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:27.769820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:27.769890 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:27.770007 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:27.771505 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.772264 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.773622 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:27.774102 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:27.774453 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:27.779671 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:27.779691 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:27.779710 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:27.779761 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:27.782966 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.783845 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.783854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.783856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.784034 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:27.784423 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:27.784427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:27.784429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:27.784807 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:27.785054 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:27.785385 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:21:27.785575 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:21:27.786708 21657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:27.836511 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.835971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42704)
0807 11:21:27.836252 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.836349 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 11:21:27.837862 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:27.837322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42710)
0807 11:21:27.837538 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:27.837625 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 11:21:27.882318 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21529
I20260807 11:21:27.906812 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21139
I20260807 11:21:27.908214 21207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:28.025936 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:28.025230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:43109 (local address 127.0.0.1:42724)
0807 11:21:28.025452 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:28.025583 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 11:21:28.026010 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21139
2026-08-07T11:21:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5212 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T11:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:28Z Disabled control of system clock
I20260807 11:21:28.141682 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40835
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:28.521320 21721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:28.521672 21721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:28.521760 21721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:28.529539 21721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:28.529675 21721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:28.529714 21721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:28.529767 21721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:28.529814 21721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:28.539815 21721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:28.541237 21721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:28.543229 21721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:28.556279 21727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:28.556337 21728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:28.556314 21730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:28.557463 21721 server_base.cc:1034] running on GCE node
I20260807 11:21:28.558573 21721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:28.560413 21721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:28.561733 21721 hybrid_clock.cc:648] HybridClock initialized: now 1786101688561692 us; error 81 us; skew 500 ppm
I20260807 11:21:28.562183 21721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:28.565264 21721 webserver.cc:460] Webserver started at http://127.2.223.190:34629/ using document root <none> and password file <none>
I20260807 11:21:28.566159 21721 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:28.566330 21721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:28.566787 21721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:28.570255 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "019099311809441fb0ac23e7629ae0c5"
format_stamp: "Formatted at 2026-08-07 11:21:28 on dist-test-slave-5l3k"
server_key: "00d369b0904291fc790fc0e5fb4d2c66"
server_key_iv: "099727b9859c5ad3e80edb25e72b8191"
server_key_version: "encryptionkey@0"
I20260807 11:21:28.571372 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "019099311809441fb0ac23e7629ae0c5"
format_stamp: "Formatted at 2026-08-07 11:21:28 on dist-test-slave-5l3k"
server_key: "00d369b0904291fc790fc0e5fb4d2c66"
server_key_iv: "099727b9859c5ad3e80edb25e72b8191"
server_key_version: "encryptionkey@0"
I20260807 11:21:28.577653 21721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:28.582000 21736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:28.583649 21721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:28.583907 21721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "019099311809441fb0ac23e7629ae0c5"
format_stamp: "Formatted at 2026-08-07 11:21:28 on dist-test-slave-5l3k"
server_key: "00d369b0904291fc790fc0e5fb4d2c66"
server_key_iv: "099727b9859c5ad3e80edb25e72b8191"
server_key_version: "encryptionkey@0"
I20260807 11:21:28.584184 21721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:28.607437 21721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:28.608431 21721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:28.608771 21721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:28.629446 21721 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:40835
I20260807 11:21:28.629472 21789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:40835 every 8 connection(s)
I20260807 11:21:28.631129 21721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:28.635579 21790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:28.640842 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21721
I20260807 11:21:28.641008 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:28.641538 2942 external_mini_cluster.cc:1428] Setting key 2af9439aba68bbd65325eacfd167064c
I20260807 11:21:28.647853 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: Bootstrap starting.
I20260807 11:21:28.652190 21790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:28.653620 21790 log.cc:824] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:28.657585 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: No bootstrap required, opened a new log
I20260807 11:21:28.658131 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:28.643788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.2.223.190:40835)
0807 11:21:28.645075 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:28.645092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:28.645146 (+ 54us) server_negotiation.cc:407] Connection header received
0807 11:21:28.647144 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:28.647172 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:28.647828 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:28.648275 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:28.649024 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:28.650445 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:28.652812 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:28.653629 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:28.653829 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:28.655318 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:28.655331 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:28.655358 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:28.656008 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:28.656099 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:28.656124 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:28.656226 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:28.656524 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 11:21:28.656930 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":484,"threads_started":1}
I20260807 11:21:28.665900 21790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } }
I20260807 11:21:28.666389 21790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:28.666568 21790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019099311809441fb0ac23e7629ae0c5, State: Initialized, Role: FOLLOWER
I20260807 11:21:28.667248 21790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } }
I20260807 11:21:28.667548 21790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:28.667732 21790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:28.667960 21790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:28.670538 21790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } }
I20260807 11:21:28.671219 21790 leader_election.cc:302] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 019099311809441fb0ac23e7629ae0c5; no voters:
I20260807 11:21:28.671952 21790 leader_election.cc:288] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:28.672181 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:28.673009 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 019099311809441fb0ac23e7629ae0c5, State: Running, Role: LEADER
I20260807 11:21:28.673767 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } }
I20260807 11:21:28.674561 21790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:28.677270 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "019099311809441fb0ac23e7629ae0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } } }
I20260807 11:21:28.677372 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 019099311809441fb0ac23e7629ae0c5. Latest consensus state: current_term: 1 leader_uuid: "019099311809441fb0ac23e7629ae0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019099311809441fb0ac23e7629ae0c5" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40835 } } }
I20260807 11:21:28.678054 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:28.678056 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:28.680514 21803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:28.687129 21803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:28.698467 21803 catalog_manager.cc:1347] Generated new cluster ID: 5cccdd54021b4a368cf75823cc4ab59c
I20260807 11:21:28.698658 21803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:28.715173 21803 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:28.717298 21803 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:28.725705 21803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: Generated new TSK 0
I20260807 11:21:28.726747 21803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:28.742725 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--builtin_ntp_servers=127.2.223.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:29.190536 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:29.191056 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:29.191252 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:29.199054 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:29.199182 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:29.199354 21814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:29.209189 21814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:29.210795 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:29.212842 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:29.228637 21820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:29.228601 21821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:29.228660 21823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:29.229841 21814 server_base.cc:1034] running on GCE node
I20260807 11:21:29.230671 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:29.231792 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:29.233009 21814 hybrid_clock.cc:648] HybridClock initialized: now 1786101689232929 us; error 82 us; skew 500 ppm
I20260807 11:21:29.233428 21814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:29.236235 21814 webserver.cc:460] Webserver started at http://127.2.223.129:34755/ using document root <none> and password file <none>
I20260807 11:21:29.237051 21814 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:29.237218 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:29.237586 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:29.240866 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "266e3451ffa344f9aa58a3c5d35ce285"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "731d1f4934b909ac6ef8fa45b9352be1"
server_key_iv: "bf3ddd318664b31c4e7f8fb5b1ec64de"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.241779 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "266e3451ffa344f9aa58a3c5d35ce285"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "731d1f4934b909ac6ef8fa45b9352be1"
server_key_iv: "bf3ddd318664b31c4e7f8fb5b1ec64de"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.247713 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:21:29.251775 21830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.253422 21814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:21:29.253633 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "266e3451ffa344f9aa58a3c5d35ce285"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "731d1f4934b909ac6ef8fa45b9352be1"
server_key_iv: "bf3ddd318664b31c4e7f8fb5b1ec64de"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.253896 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:29.268859 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:29.269827 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:29.270182 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:29.271385 21814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:29.273029 21814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:29.273178 21814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.273313 21814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:29.273403 21814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.309641 21814 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:39321
I20260807 11:21:29.309692 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:39321 every 8 connection(s)
I20260807 11:21:29.311086 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:29.314005 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21814
I20260807 11:21:29.314188 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:29.314831 2942 external_mini_cluster.cc:1428] Setting key 593735631e93238644d2d06f931f01cb
I20260807 11:21:29.318140 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--builtin_ntp_servers=127.2.223.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:29.326256 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:29.314212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:51815 (local address 127.2.223.190:40835)
0807 11:21:29.314795 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:29.314799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:29.316385 (+ 1586us) server_negotiation.cc:407] Connection header received
0807 11:21:29.318699 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:29.318706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:29.318805 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:29.318932 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:29.320092 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.321251 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.322969 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.323679 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.323832 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:29.325221 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:29.325230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:29.325237 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:29.325267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:29.325323 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:29.325341 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:29.325444 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:29.325680 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 11:21:29.325803 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":319,"threads_started":1}
I20260807 11:21:29.327188 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:29.314954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40835 (local address 127.2.223.129:51815)
0807 11:21:29.316180 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:29.316230 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:29.318350 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:29.319174 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:29.319182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:29.319277 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:29.319888 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:29.319905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.321429 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.321433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:29.322820 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:29.322834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.323901 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.323906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:29.324117 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:29.324868 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:29.324871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:29.324920 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:29.325005 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:29.325452 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:29.325461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:29.325468 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:21:29.325624 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 11:21:29.325948 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":380,"threads_started":1}
I20260807 11:21:29.328861 21948 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40835
I20260807 11:21:29.329201 21948 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:29.329996 21948 heartbeater.cc:511] Master 127.2.223.190:40835 requested a full tablet report, sending...
I20260807 11:21:29.332247 21754 ts_manager.cc:194] Registered new tserver with Master: 266e3451ffa344f9aa58a3c5d35ce285 (127.2.223.129:39321)
I20260807 11:21:29.333778 21754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.129:51815
W20260807 11:21:29.694262 21952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:29.694684 21952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:29.694873 21952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:29.702463 21952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:29.702670 21952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:29.702870 21952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:29.712718 21952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:29.714759 21952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:29.717018 21952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:29.731642 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:29.731763 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:29.731731 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:29.732955 21952 server_base.cc:1034] running on GCE node
I20260807 11:21:29.733719 21952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:29.734877 21952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:29.736104 21952 hybrid_clock.cc:648] HybridClock initialized: now 1786101689736026 us; error 65 us; skew 500 ppm
I20260807 11:21:29.736462 21952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:29.739113 21952 webserver.cc:460] Webserver started at http://127.2.223.130:39007/ using document root <none> and password file <none>
I20260807 11:21:29.739920 21952 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:29.740047 21952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:29.740428 21952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:29.744089 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "2acad2445c124f259276a9c1c3d56cb2"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "383ecf08213a326c8bf443e97c361023"
server_key_iv: "232d930cac866bfea107faa1974e5f1a"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.744962 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "2acad2445c124f259276a9c1c3d56cb2"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "383ecf08213a326c8bf443e97c361023"
server_key_iv: "232d930cac866bfea107faa1974e5f1a"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.750629 21952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 11:21:29.754418 21967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.756023 21952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:21:29.756268 21952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "2acad2445c124f259276a9c1c3d56cb2"
format_stamp: "Formatted at 2026-08-07 11:21:29 on dist-test-slave-5l3k"
server_key: "383ecf08213a326c8bf443e97c361023"
server_key_iv: "232d930cac866bfea107faa1974e5f1a"
server_key_version: "encryptionkey@0"
I20260807 11:21:29.756592 21952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:29.781517 21952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:29.782684 21952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:29.783025 21952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:29.784086 21952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:29.785725 21952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:29.785868 21952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.786017 21952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:29.786119 21952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:29.822996 21952 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:39255
I20260807 11:21:29.823051 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:39255 every 8 connection(s)
I20260807 11:21:29.824455 21952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:29.827591 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 21952
I20260807 11:21:29.827821 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:29.828720 2942 external_mini_cluster.cc:1428] Setting key 1214e5220b101846a1de69c3561c3a09
I20260807 11:21:29.832772 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--builtin_ntp_servers=127.2.223.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:29.840643 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:29.827708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49883 (local address 127.2.223.190:40835)
0807 11:21:29.830089 (+ 2381us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:29.830095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:29.830119 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:21:29.831898 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:29.831903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:29.831989 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:29.832101 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:29.833716 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.834675 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.837338 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.837773 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.837885 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:29.839797 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:29.839802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:29.839805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:29.839829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:29.839873 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:29.839887 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:29.839964 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:29.840261 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 11:21:29.840358 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2251,"thread_start_us":2136,"threads_started":1}
I20260807 11:21:29.842345 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:29.828334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40835 (local address 127.2.223.130:49883)
0807 11:21:29.829484 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:29.829544 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:29.831559 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:29.832426 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:29.832441 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:29.832578 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:29.833495 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:29.833524 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.834877 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.834885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:29.837163 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:29.837183 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:29.838036 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:29.838046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:29.838423 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:29.839350 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:29.839356 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:29.839420 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:29.839562 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:29.839996 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:29.840010 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:29.840021 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:21:29.840242 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 11:21:29.840753 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":372,"threads_started":1}
I20260807 11:21:29.844221 22084 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40835
I20260807 11:21:29.844575 22084 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:29.845613 22084 heartbeater.cc:511] Master 127.2.223.190:40835 requested a full tablet report, sending...
I20260807 11:21:29.847487 21754 ts_manager.cc:194] Registered new tserver with Master: 2acad2445c124f259276a9c1c3d56cb2 (127.2.223.130:39255)
I20260807 11:21:29.848502 21754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.130:49883
W20260807 11:21:30.205090 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:30.205466 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:30.205647 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:30.214061 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:30.214236 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:30.214406 22089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:30.224272 22089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:30.225696 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:30.227674 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:30.241336 22095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:30.241528 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:30.241483 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:30.242597 22089 server_base.cc:1034] running on GCE node
I20260807 11:21:30.243296 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:30.244385 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:30.245658 22089 hybrid_clock.cc:648] HybridClock initialized: now 1786101690245604 us; error 43 us; skew 500 ppm
I20260807 11:21:30.246031 22089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:30.248880 22089 webserver.cc:460] Webserver started at http://127.2.223.131:42471/ using document root <none> and password file <none>
I20260807 11:21:30.249752 22089 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:30.249907 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:30.250286 22089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:30.253765 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "da43359f56dc4af585090a4e31cb1734"
format_stamp: "Formatted at 2026-08-07 11:21:30 on dist-test-slave-5l3k"
server_key: "73cc1993d9c31aeeb76eccb674e9767c"
server_key_iv: "3d04a4581e0bb1d7210d26ca1886d99b"
server_key_version: "encryptionkey@0"
I20260807 11:21:30.254920 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "da43359f56dc4af585090a4e31cb1734"
format_stamp: "Formatted at 2026-08-07 11:21:30 on dist-test-slave-5l3k"
server_key: "73cc1993d9c31aeeb76eccb674e9767c"
server_key_iv: "3d04a4581e0bb1d7210d26ca1886d99b"
server_key_version: "encryptionkey@0"
I20260807 11:21:30.261087 22089 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:30.265311 22105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:30.266908 22089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:30.267115 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "da43359f56dc4af585090a4e31cb1734"
format_stamp: "Formatted at 2026-08-07 11:21:30 on dist-test-slave-5l3k"
server_key: "73cc1993d9c31aeeb76eccb674e9767c"
server_key_iv: "3d04a4581e0bb1d7210d26ca1886d99b"
server_key_version: "encryptionkey@0"
I20260807 11:21:30.267395 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:30.289306 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:30.290300 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:30.290694 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:30.291857 22089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:30.293596 22089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:30.293736 22089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:30.293890 22089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:30.293988 22089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:30.332345 22089 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:35969
I20260807 11:21:30.332404 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:35969 every 8 connection(s)
I20260807 11:21:30.333835 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:30.337004 21948 heartbeater.cc:503] Master 127.2.223.190:40835 was elected leader, sending a full tablet report...
I20260807 11:21:30.343727 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22089
I20260807 11:21:30.343936 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:30.344424 2942 external_mini_cluster.cc:1428] Setting key 59e633b9f3e930c49d44e69c5ec35c56
I20260807 11:21:30.348519 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.336835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:47105 (local address 127.2.223.190:40835)
0807 11:21:30.337026 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.337030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.339013 (+ 1983us) server_negotiation.cc:407] Connection header received
0807 11:21:30.341079 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.341084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.341167 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.341262 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:30.342471 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.343366 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.345349 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.345944 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.346074 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.347790 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:30.347795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:30.347799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:30.347818 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:30.347854 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:30.347866 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:30.347940 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.348177 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.348266 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:21:30.349707 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.337504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:40835 (local address 127.2.223.131:47105)
0807 11:21:30.338831 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.338892 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.340760 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.341501 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.341510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.341611 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:30.342275 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.342295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.343545 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.343550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.345212 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.345227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.346219 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.346228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.346589 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.347383 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:30.347387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:30.347452 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:30.347579 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:30.347955 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:30.347964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:30.347970 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.348138 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.348490 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":432,"threads_started":1}
I20260807 11:21:30.351260 22223 heartbeater.cc:347] Connected to a master server at 127.2.223.190:40835
I20260807 11:21:30.351581 22223 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:30.352470 22223 heartbeater.cc:511] Master 127.2.223.190:40835 requested a full tablet report, sending...
I20260807 11:21:30.354143 21754 ts_manager.cc:194] Registered new tserver with Master: da43359f56dc4af585090a4e31cb1734 (127.2.223.131:35969)
I20260807 11:21:30.355051 21754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.131:47105
I20260807 11:21:30.360316 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:30.371729 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.365118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.2.223.190:40835)
0807 11:21:30.365371 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.365379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.365739 (+ 360us) server_negotiation.cc:407] Connection header received
0807 11:21:30.365974 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.365978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.366051 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.366165 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:30.366802 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.367564 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.369087 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.369545 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.369709 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.370862 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:30.370867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:30.370870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:30.370893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:30.370929 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:30.370940 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:30.371038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.371235 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.371345 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:21:30.376207 21754 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:30.383658 21754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:30.411957 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.396994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:39255 (local address 127.0.0.1:49984)
0807 11:21:30.398092 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.398160 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.398353 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.400118 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.400126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.400172 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:30.400684 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.400703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.403669 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.403679 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.407446 (+ 3767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.407477 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.410532 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.410545 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.410896 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.410984 (+ 88us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.411161 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.411527 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":9,"thread_start_us":581,"threads_started":1}
I20260807 11:21:30.413010 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.397678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.2.223.130:39255)
0807 11:21:30.399408 (+ 1730us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.399419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.399452 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:21:30.399561 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.399572 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.399862 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.400963 (+ 1101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:30.401258 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.403431 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.407879 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.410115 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.412038 (+ 1923us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.412335 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.412426 (+ 91us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.412657 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":1454,"threads_started":1}
I20260807 11:21:30.413772 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.398158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.2.223.131:35969)
0807 11:21:30.399103 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.399114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.399796 (+ 682us) server_negotiation.cc:407] Connection header received
0807 11:21:30.399910 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.399918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.400198 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.400444 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:30.401659 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.404356 (+ 2697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.408216 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.410332 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.410541 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.410980 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.413173 (+ 2193us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.413386 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":610,"threads_started":1}
I20260807 11:21:30.414717 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.397333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:35969 (local address 127.0.0.1:36068)
0807 11:21:30.399552 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.399575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.399701 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.400804 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.400811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.400851 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:30.401357 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.401373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.404592 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.404601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.407859 (+ 3258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.407887 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.411185 (+ 3298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.411195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.412898 (+ 1703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.412934 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.414042 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.414233 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2101,"mutex_wait_us":1391,"thread_start_us":1937,"threads_started":1}
I20260807 11:21:30.417847 22017 tablet_service.cc:1467] Processing CreateTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 (DEFAULT_TABLE table=test-table [id=0d2c983eddee4895b9159e59fa961c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:30.419373 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85dd79a8b1c4e01bb5910561e65f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:30.418812 22155 tablet_service.cc:1467] Processing CreateTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 (DEFAULT_TABLE table=test-table [id=0d2c983eddee4895b9159e59fa961c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:30.420279 22155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85dd79a8b1c4e01bb5910561e65f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:30.421108 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.399946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34522)
0807 11:21:30.412623 (+ 12677us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.412648 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.412797 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.413984 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.413988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.414018 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:30.414396 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.414410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.416036 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.416042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.418163 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.418180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.420229 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.420237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.420468 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.420504 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.420615 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.420767 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12535,"thread_start_us":12376,"threads_started":1}
I20260807 11:21:30.422096 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.400109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.2.223.129:39321)
0807 11:21:30.412100 (+ 11991us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.412113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.412944 (+ 831us) server_negotiation.cc:407] Connection header received
0807 11:21:30.413060 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.413069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.413403 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.413608 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:30.414616 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.415851 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.418399 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.419969 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.420133 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.420435 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.421525 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.421706 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11722,"thread_start_us":11536,"threads_started":1}
I20260807 11:21:30.426147 21880 tablet_service.cc:1467] Processing CreateTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 (DEFAULT_TABLE table=test-table [id=0d2c983eddee4895b9159e59fa961c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:30.427860 21880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85dd79a8b1c4e01bb5910561e65f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:30.432554 22242 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Bootstrap starting.
I20260807 11:21:30.435438 22243 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Bootstrap starting.
I20260807 11:21:30.438829 22242 tablet_bootstrap.cc:654] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:30.439402 22243 tablet_bootstrap.cc:654] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:30.440326 22242 log.cc:824] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:30.441004 22243 log.cc:824] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:30.444681 22245 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Bootstrap starting.
I20260807 11:21:30.445957 22243 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: No bootstrap required, opened a new log
I20260807 11:21:30.446282 22243 ts_tablet_manager.cc:1397] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260807 11:21:30.446326 22242 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: No bootstrap required, opened a new log
I20260807 11:21:30.446693 22242 ts_tablet_manager.cc:1397] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260807 11:21:30.449177 22245 tablet_bootstrap.cc:654] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:30.450598 22245 log.cc:824] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:30.454690 22245 tablet_bootstrap.cc:492] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: No bootstrap required, opened a new log
I20260807 11:21:30.455044 22245 ts_tablet_manager.cc:1397] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 11:21:30.455310 22242 raft_consensus.cc:348] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.455753 22242 raft_consensus.cc:374] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:30.455866 22242 raft_consensus.cc:729] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acad2445c124f259276a9c1c3d56cb2, State: Initialized, Role: FOLLOWER
I20260807 11:21:30.456568 22242 consensus_queue.cc:260] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.457590 22243 raft_consensus.cc:348] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.458137 22243 raft_consensus.cc:374] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:30.458245 22243 raft_consensus.cc:729] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da43359f56dc4af585090a4e31cb1734, State: Initialized, Role: FOLLOWER
I20260807 11:21:30.459103 22243 consensus_queue.cc:260] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.466813 22223 heartbeater.cc:503] Master 127.2.223.190:40835 was elected leader, sending a full tablet report...
I20260807 11:21:30.467237 22084 heartbeater.cc:503] Master 127.2.223.190:40835 was elected leader, sending a full tablet report...
I20260807 11:21:30.468115 22242 ts_tablet_manager.cc:1428] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260807 11:21:30.468917 22245 raft_consensus.cc:348] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.469389 22243 ts_tablet_manager.cc:1428] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Time spent starting tablet: real 0.023s user 0.018s sys 0.003s
I20260807 11:21:30.469564 22245 raft_consensus.cc:374] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:30.469743 22245 raft_consensus.cc:729] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266e3451ffa344f9aa58a3c5d35ce285, State: Initialized, Role: FOLLOWER
I20260807 11:21:30.470566 22245 consensus_queue.cc:260] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.473902 22245 ts_tablet_manager.cc:1428] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260807 11:21:30.558314 22248 raft_consensus.cc:482] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:30.558779 22248 raft_consensus.cc:504] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.560751 22248 leader_election.cc:288] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da43359f56dc4af585090a4e31cb1734 (127.2.223.131:35969), 266e3451ffa344f9aa58a3c5d35ce285 (127.2.223.129:39321)
W20260807 11:21:30.564145 21949 tablet.cc:2369] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:30.569340 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.562313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:33221 (local address 127.2.223.129:39321)
0807 11:21:30.562574 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.562591 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.563232 (+ 641us) server_negotiation.cc:407] Connection header received
0807 11:21:30.563409 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.563413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.563484 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.563595 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:30.564113 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.564966 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.567155 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.568338 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.568522 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.568614 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.568956 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.569087 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:21:30.569394 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.561204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:48979 (local address 127.2.223.131:35969)
0807 11:21:30.561432 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.561440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.562210 (+ 770us) server_negotiation.cc:407] Connection header received
0807 11:21:30.562618 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.562628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.562755 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.562857 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:30.563615 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.564750 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.567277 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.568356 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.568547 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.568631 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.569005 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.569150 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 11:21:30.569437 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.562172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.2.223.130:33221)
0807 11:21:30.563096 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.563117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.563264 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.563628 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.563632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.563663 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:30.563949 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.563956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.565152 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.565163 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.566897 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.566920 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.568570 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.568578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.568760 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.568789 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.568901 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.569050 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":60,"thread_start_us":528,"threads_started":1}
I20260807 11:21:30.570022 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.561198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:35969 (local address 127.2.223.130:48979)
0807 11:21:30.562068 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.562099 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:30.562339 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:30.562931 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:30.562936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:30.562967 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:30.563461 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.563472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.564919 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.564924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.566922 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:30.566937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.568619 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.568637 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:30.568821 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.568845 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:21:30.568956 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:21:30.569095 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":89,"thread_start_us":430,"threads_started":1}
I20260807 11:21:30.571178 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85dd79a8b1c4e01bb5910561e65f337" candidate_uuid: "2acad2445c124f259276a9c1c3d56cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da43359f56dc4af585090a4e31cb1734" is_pre_election: true
I20260807 11:21:30.571220 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85dd79a8b1c4e01bb5910561e65f337" candidate_uuid: "2acad2445c124f259276a9c1c3d56cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266e3451ffa344f9aa58a3c5d35ce285" is_pre_election: true
I20260807 11:21:30.571882 21902 raft_consensus.cc:2454] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acad2445c124f259276a9c1c3d56cb2 in term 0.
I20260807 11:21:30.571882 22176 raft_consensus.cc:2454] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acad2445c124f259276a9c1c3d56cb2 in term 0.
I20260807 11:21:30.572620 21972 leader_election.cc:302] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 266e3451ffa344f9aa58a3c5d35ce285, 2acad2445c124f259276a9c1c3d56cb2; no voters:
I20260807 11:21:30.573022 22248 raft_consensus.cc:2781] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:30.573199 22248 raft_consensus.cc:482] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:30.573319 22248 raft_consensus.cc:3037] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:30.575930 22248 raft_consensus.cc:504] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
I20260807 11:21:30.576638 22248 leader_election.cc:288] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [CANDIDATE]: Term 1 election: Requested vote from peers da43359f56dc4af585090a4e31cb1734 (127.2.223.131:35969), 266e3451ffa344f9aa58a3c5d35ce285 (127.2.223.129:39321)
W20260807 11:21:30.576938 22085 tablet.cc:2369] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:30.577210 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85dd79a8b1c4e01bb5910561e65f337" candidate_uuid: "2acad2445c124f259276a9c1c3d56cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da43359f56dc4af585090a4e31cb1734"
I20260807 11:21:30.577251 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85dd79a8b1c4e01bb5910561e65f337" candidate_uuid: "2acad2445c124f259276a9c1c3d56cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266e3451ffa344f9aa58a3c5d35ce285"
I20260807 11:21:30.577451 22176 raft_consensus.cc:3037] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:30.577512 21902 raft_consensus.cc:3037] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:30.579906 22176 raft_consensus.cc:2454] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acad2445c124f259276a9c1c3d56cb2 in term 1.
I20260807 11:21:30.580051 21902 raft_consensus.cc:2454] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acad2445c124f259276a9c1c3d56cb2 in term 1.
I20260807 11:21:30.580415 21972 leader_election.cc:302] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acad2445c124f259276a9c1c3d56cb2, da43359f56dc4af585090a4e31cb1734; no voters:
I20260807 11:21:30.580793 22248 raft_consensus.cc:2781] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:30.581277 22248 raft_consensus.cc:686] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 2acad2445c124f259276a9c1c3d56cb2, State: Running, Role: LEADER
I20260807 11:21:30.581820 22248 consensus_queue.cc:237] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } }
W20260807 11:21:30.585819 22224 tablet.cc:2369] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:30.586910 21753 catalog_manager.cc:5495] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2acad2445c124f259276a9c1c3d56cb2 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "2acad2445c124f259276a9c1c3d56cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acad2445c124f259276a9c1c3d56cb2" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 39255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 } health_report { overall_health: UNKNOWN } } }
I20260807 11:21:30.617924 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.612547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.2.223.130:39255)
0807 11:21:30.612745 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.612749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.612768 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:21:30.612862 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.612866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.612964 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.613116 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:30.613431 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.614158 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.615785 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.616230 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.616341 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.616824 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:30.617380 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:30.617495 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.617618 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.617689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:21:30.626793 22176 raft_consensus.cc:1261] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 1 FOLLOWER]: Refusing update from remote peer 2acad2445c124f259276a9c1c3d56cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:30.626792 21902 raft_consensus.cc:1261] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 1 FOLLOWER]: Refusing update from remote peer 2acad2445c124f259276a9c1c3d56cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:30.627875 22253 consensus_queue.cc:1035] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da43359f56dc4af585090a4e31cb1734" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:30.628247 22248 consensus_queue.cc:1035] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266e3451ffa344f9aa58a3c5d35ce285" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:30.646853 22259 mvcc.cc:205] Tried to move back new op lower bound from 7315872524796317696 to 7315872524628131840. Current Snapshot: MvccSnapshot[applied={T|T < 7315872524796317696}]
I20260807 11:21:30.671622 21752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:30.671998 21752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:30.676389 21752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 019099311809441fb0ac23e7629ae0c5: Sending DeleteTablet for 3 replicas of tablet b85dd79a8b1c4e01bb5910561e65f337
I20260807 11:21:30.678092 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:30 UTC) from {username='slave'} at 127.0.0.1:49984
I20260807 11:21:30.678092 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:30 UTC) from {username='slave'} at 127.0.0.1:34522
I20260807 11:21:30.679239 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21814
I20260807 11:21:30.679242 22267 tablet_replica.cc:330] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: stopping tablet replica
I20260807 11:21:30.679391 22155 tablet_service.cc:1514] Processing DeleteTablet for tablet b85dd79a8b1c4e01bb5910561e65f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:30 UTC) from {username='slave'} at 127.0.0.1:36068
I20260807 11:21:30.679772 22267 raft_consensus.cc:2229] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:30.680380 22267 raft_consensus.cc:2258] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:30.684396 22267 ts_tablet_manager.cc:1905] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:30.693120 22267 ts_tablet_manager.cc:1918] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:30.693822 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.680974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.2.223.129:39321)
0807 11:21:30.681219 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.681228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.683021 (+ 1793us) server_negotiation.cc:407] Connection header received
0807 11:21:30.683241 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.683250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.683388 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.683516 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:30.684365 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.685784 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.688333 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.689238 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.689412 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.691396 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:30.691413 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:30.691440 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:30.692490 (+ 1050us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:30.692594 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:30.692632 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:30.692759 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.693082 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.693283 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:21:30.693485 22267 log.cc:1196] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/b85dd79a8b1c4e01bb5910561e65f337
I20260807 11:21:30.694940 21943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:30.696072 22269 tablet_replica.cc:330] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: stopping tablet replica
I20260807 11:21:30.696713 22269 raft_consensus.cc:2229] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:30.697491 22268 tablet_replica.cc:330] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: stopping tablet replica
I20260807 11:21:30.698854 22268 raft_consensus.cc:2229] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:30.699709 22268 raft_consensus.cc:2258] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:30.698149 22269 raft_consensus.cc:2258] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:30.703368 22269 ts_tablet_manager.cc:1905] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:30.704298 22268 ts_tablet_manager.cc:1905] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:30.709811 22268 ts_tablet_manager.cc:1918] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:30.710071 22268 log.cc:1196] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/b85dd79a8b1c4e01bb5910561e65f337
I20260807 11:21:30.710824 22268 ts_tablet_manager.cc:1939] T b85dd79a8b1c4e01bb5910561e65f337 P 2acad2445c124f259276a9c1c3d56cb2: Deleting consensus metadata
I20260807 11:21:30.711100 22269 ts_tablet_manager.cc:1918] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:30.711354 22269 log.cc:1196] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/b85dd79a8b1c4e01bb5910561e65f337
I20260807 11:21:30.711982 22269 ts_tablet_manager.cc:1939] T b85dd79a8b1c4e01bb5910561e65f337 P da43359f56dc4af585090a4e31cb1734: Deleting consensus metadata
I20260807 11:21:30.713179 21738 catalog_manager.cc:4841] TS 2acad2445c124f259276a9c1c3d56cb2 (127.2.223.130:39255): tablet b85dd79a8b1c4e01bb5910561e65f337 (table test-table [id=0d2c983eddee4895b9159e59fa961c15]) successfully deleted
I20260807 11:21:30.713960 21740 catalog_manager.cc:4841] TS da43359f56dc4af585090a4e31cb1734 (127.2.223.131:35969): tablet b85dd79a8b1c4e01bb5910561e65f337 (table test-table [id=0d2c983eddee4895b9159e59fa961c15]) successfully deleted
I20260807 11:21:30.801811 22267 ts_tablet_manager.cc:1939] T b85dd79a8b1c4e01bb5910561e65f337 P 266e3451ffa344f9aa58a3c5d35ce285: Deleting consensus metadata
I20260807 11:21:30.802588 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21814
W20260807 11:21:30.826638 21740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.223.129:39321 (error 108)
I20260807 11:21:30.827149 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21952
I20260807 11:21:30.842111 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.829282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.2.223.130:39255)
0807 11:21:30.829569 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.829578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.829606 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:21:30.829693 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.829701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.829884 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.830020 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:30.831221 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.832923 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.835787 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.837584 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:30.837805 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:30.839883 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:30.839901 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:30.839930 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:30.841122 (+ 1192us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:30.841202 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:30.841227 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:30.841364 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:30.841588 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:21:30.841716 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 11:21:30.842716 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.842183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34534)
0807 11:21:30.842372 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.842536 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 11:21:30.843212 22079 generic_service.cc:196] Checking for leaks (request via RPC)
W20260807 11:21:30.843166 21740 catalog_manager.cc:4584] TS 266e3451ffa344f9aa58a3c5d35ce285 (127.2.223.129:39321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b85dd79a8b1c4e01bb5910561e65f337: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
I20260807 11:21:30.897063 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.896395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34538)
0807 11:21:30.896700 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.896825 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 11:21:30.898303 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.897871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34540)
0807 11:21:30.898057 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.898153 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 11:21:30.959026 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.958404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34546)
0807 11:21:30.958704 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.958826 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260807 11:21:30.960297 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.959876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34560)
0807 11:21:30.960072 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:30.960159 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 11:21:30.970906 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21952
I20260807 11:21:30.997040 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22089
I20260807 11:21:31.004386 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:30.997896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.2.223.131:35969)
0807 11:21:30.998079 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:30.998083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:30.998125 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:21:30.998281 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:30.998286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:30.998374 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:30.998462 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:30.998874 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:30.999600 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:31.001006 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:31.001393 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:31.001484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:31.003012 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:31.003023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:31.003038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:31.003682 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:31.003737 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:31.003760 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:31.003856 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:31.003989 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 11:21:31.004116 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:21:31.005194 22218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:31.041440 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:31.040720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34576)
0807 11:21:31.041050 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:31.041176 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 11:21:31.042780 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:31.042255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34578)
0807 11:21:31.042517 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:31.042628 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260807 11:21:31.099133 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22089
I20260807 11:21:31.121632 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 21721
I20260807 11:21:31.122929 21784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:31.239655 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:31.238083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:39321 (local address 127.0.0.1:34584)
0807 11:21:31.239160 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:31.239331 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":829}
I20260807 11:21:31.239977 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 21721
2026-08-07T11:21:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3207 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:31 dist-test-slave-5l3k krb5kdc[22280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:31 dist-test-slave-5l3k krb5kdc[22280](info): set up 2 sockets
Aug 07 11:21:31 dist-test-slave-5l3k krb5kdc[22280](info): commencing operation
krb5kdc: starting...
W20260807 11:21:32.468291 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.001s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:32 dist-test-slave-5l3k krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:21:32.858548 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35935
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:33.237053 22293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:33.237409 22293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:33.237506 22293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:33.245152 22293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:33.245286 22293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:33.245337 22293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:33.245397 22293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:33.245442 22293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:33.255936 22293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:33.257418 22293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:33.258898 22293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:33.272383 22298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:33.272426 22299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:33.272336 22301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:33.273373 22293 server_base.cc:1034] running on GCE node
I20260807 11:21:33.274327 22293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:33.276014 22293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:33.277308 22293 hybrid_clock.cc:648] HybridClock initialized: now 1786101693277245 us; error 49 us; skew 500 ppm
I20260807 11:21:33.277740 22293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:33 dist-test-slave-5l3k krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:33.286811 22293 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:21:33.288761 22293 webserver.cc:460] Webserver started at http://127.2.223.190:40693/ using document root <none> and password file <none>
I20260807 11:21:33.289647 22293 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:33.289834 22293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:33.290225 22293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:33.293910 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "b39dd1abcb2e499ab51180344a33e90c"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "3843530749c4ec58bb69dad1ffdd599a"
server_key_iv: "df6614a785a7495c797aaa09764400c7"
server_key_version: "encryptionkey@0"
I20260807 11:21:33.294937 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "b39dd1abcb2e499ab51180344a33e90c"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "3843530749c4ec58bb69dad1ffdd599a"
server_key_iv: "df6614a785a7495c797aaa09764400c7"
server_key_version: "encryptionkey@0"
I20260807 11:21:33.301374 22293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:33.305613 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:33.307245 22293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:33.307502 22293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "b39dd1abcb2e499ab51180344a33e90c"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "3843530749c4ec58bb69dad1ffdd599a"
server_key_iv: "df6614a785a7495c797aaa09764400c7"
server_key_version: "encryptionkey@0"
I20260807 11:21:33.307754 22293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:33.325137 22293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:33.330355 22293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:33.330763 22293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:33.351902 22293 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:35935
I20260807 11:21:33.351950 22364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:35935 every 8 connection(s)
I20260807 11:21:33.353531 22293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:33.357719 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22293
I20260807 11:21:33.357985 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:33.358543 22365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:33.358711 2942 external_mini_cluster.cc:1428] Setting key 1269792d63eec6729143f0fbd5f773b0
I20260807 11:21:33.377576 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: Bootstrap starting.
Aug 07 11:21:33 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:33.382166 22365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:33.383814 22365 log.cc:824] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:33.387917 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: No bootstrap required, opened a new log
I20260807 11:21:33.389966 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:33.361064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.2.223.190:35935)
0807 11:21:33.362252 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:33.362271 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:33.362748 (+ 477us) server_negotiation.cc:407] Connection header received
0807 11:21:33.365073 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:33.365109 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:33.365664 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:33.366088 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:33.368720 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:33.370263 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:33.372601 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:33.373557 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:33.373773 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:33.379791 (+ 6018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:33.379806 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:33.379835 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:33.379910 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:33.384285 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:33.385725 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:33.385737 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:33.385749 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:33.386011 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:33.386703 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:33.386708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:33.386710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:33.387357 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:33.387737 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:33.388258 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 11:21:33.388797 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":483,"threads_started":1}
I20260807 11:21:33.397562 22365 raft_consensus.cc:348] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } }
I20260807 11:21:33.398077 22365 raft_consensus.cc:374] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:33.398206 22365 raft_consensus.cc:729] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39dd1abcb2e499ab51180344a33e90c, State: Initialized, Role: FOLLOWER
I20260807 11:21:33.398895 22365 consensus_queue.cc:260] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } }
I20260807 11:21:33.399178 22365 raft_consensus.cc:388] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:33.399334 22365 raft_consensus.cc:482] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:33.399523 22365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:33.402273 22365 raft_consensus.cc:504] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } }
I20260807 11:21:33.402920 22365 leader_election.cc:302] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b39dd1abcb2e499ab51180344a33e90c; no voters:
I20260807 11:21:33.403587 22365 leader_election.cc:288] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:33.403771 22370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:33.404508 22370 raft_consensus.cc:686] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [term 1 LEADER]: Becoming Leader. State: Replica: b39dd1abcb2e499ab51180344a33e90c, State: Running, Role: LEADER
I20260807 11:21:33.405153 22370 consensus_queue.cc:237] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } }
I20260807 11:21:33.405982 22365 sys_catalog.cc:564] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:33.408298 22371 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b39dd1abcb2e499ab51180344a33e90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } } }
I20260807 11:21:33.408792 22371 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:33.410112 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39dd1abcb2e499ab51180344a33e90c. Latest consensus state: current_term: 1 leader_uuid: "b39dd1abcb2e499ab51180344a33e90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39dd1abcb2e499ab51180344a33e90c" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35935 } } }
I20260807 11:21:33.410580 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:33.411482 22375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:33.417557 22375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:33.428372 22375 catalog_manager.cc:1347] Generated new cluster ID: 942894db75f74842a22dc819e483d2c2
I20260807 11:21:33.428573 22375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:33.464541 22375 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:33.466732 22375 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:33.479084 22375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: Generated new TSK 0
I20260807 11:21:33.480322 22375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:21:33.566464 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:33.942711 22392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:33.943049 22392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:33.943223 22392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:33.951062 22392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:33.951200 22392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:33.951381 22392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:33.962231 22392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:33.963761 22392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:33.965219 22392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:33.980170 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:33.980237 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:33.980100 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:33.981122 22392 server_base.cc:1034] running on GCE node
I20260807 11:21:33.981786 22392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:33.982970 22392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:33.984226 22392 hybrid_clock.cc:648] HybridClock initialized: now 1786101693984191 us; error 57 us; skew 500 ppm
I20260807 11:21:33.984550 22392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:33 dist-test-slave-5l3k krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:33.988114 22392 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:21:33.989749 22392 webserver.cc:460] Webserver started at http://127.2.223.129:35141/ using document root <none> and password file <none>
I20260807 11:21:33.990597 22392 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:33.990751 22392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:33.991134 22392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:33.994431 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "43a25bef0cde43239ea31c19701eac21"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "c1e28440164833e84c587019852e4514"
server_key_iv: "7b81cdf09130d3e00a4b9a8fc4cc2715"
server_key_version: "encryptionkey@0"
I20260807 11:21:33.995417 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "43a25bef0cde43239ea31c19701eac21"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "c1e28440164833e84c587019852e4514"
server_key_iv: "7b81cdf09130d3e00a4b9a8fc4cc2715"
server_key_version: "encryptionkey@0"
I20260807 11:21:34.001426 22392 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:21:34.005494 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.007263 22392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:34.007475 22392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "43a25bef0cde43239ea31c19701eac21"
format_stamp: "Formatted at 2026-08-07 11:21:33 on dist-test-slave-5l3k"
server_key: "c1e28440164833e84c587019852e4514"
server_key_iv: "7b81cdf09130d3e00a4b9a8fc4cc2715"
server_key_version: "encryptionkey@0"
I20260807 11:21:34.007735 22392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:34.028214 22392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:34.033196 22392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:34.033530 22392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:34.034855 22392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:34.036408 22392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:34.036490 22392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.036643 22392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:34.036708 22392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.072853 22392 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:42881
I20260807 11:21:34.072903 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:42881 every 8 connection(s)
I20260807 11:21:34.074321 22392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:34.075052 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22392
I20260807 11:21:34.075287 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:34.075944 2942 external_mini_cluster.cc:1428] Setting key ebc8ae6a3c6219c266725a33af046f3e
Aug 07 11:21:34 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:34.100798 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:34.077775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:38551 (local address 127.2.223.190:35935)
0807 11:21:34.078297 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:34.078302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:34.079748 (+ 1446us) server_negotiation.cc:407] Connection header received
0807 11:21:34.081783 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:34.081792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:34.081896 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:34.082031 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:34.083784 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.084969 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.087892 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.088345 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.088549 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:34.093784 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:34.093794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:34.093802 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:34.093859 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:34.096927 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:34.098120 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:34.098125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:34.098127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:34.098267 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:34.098850 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:34.098853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:34.098855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:34.099144 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:34.099264 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:34.100292 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0807 11:21:34.100488 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
I20260807 11:21:34.102613 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:34.078606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35935 (local address 127.2.223.129:38551)
0807 11:21:34.079516 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:34.079582 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:34.081454 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:34.082264 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:34.082274 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:34.082947 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:34.083568 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:34.083586 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.085221 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.085231 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:34.087651 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:34.087677 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.088706 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.088716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:34.089088 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:34.090009 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:34.090055 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:34.093417 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:34.097132 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:34.097146 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:34.097171 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:34.097962 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:34.098386 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:34.098393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:34.098399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:34.098727 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:34.099292 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:34.099303 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:34.099857 (+ 554us) client_negotiation.cc:769] Sending connection context
0807 11:21:34.100206 (+ 349us) client_negotiation.cc:240] Negotiation successful
0807 11:21:34.100908 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":372,"threads_started":1}
I20260807 11:21:34.104597 22530 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35935
I20260807 11:21:34.104956 22530 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:34.105814 22530 heartbeater.cc:511] Master 127.2.223.190:35935 requested a full tablet report, sending...
I20260807 11:21:34.108692 22328 ts_manager.cc:194] Registered new tserver with Master: 43a25bef0cde43239ea31c19701eac21 (127.2.223.129:42881)
I20260807 11:21:34.111076 22328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:38551
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:21:34.162231 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:34.539186 22536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:34.539500 22536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:34.539665 22536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:34.547222 22536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:34.547346 22536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:34.547534 22536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:34.557922 22536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:34.559446 22536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:34.560878 22536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:34.575063 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:34.575637 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:34.575870 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:34.576867 22536 server_base.cc:1034] running on GCE node
I20260807 11:21:34.577553 22536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:34.578675 22536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:34.579929 22536 hybrid_clock.cc:648] HybridClock initialized: now 1786101694579876 us; error 38 us; skew 500 ppm
I20260807 11:21:34.580294 22536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:34 dist-test-slave-5l3k krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:34.583827 22536 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:21:34.585547 22536 webserver.cc:460] Webserver started at http://127.2.223.130:45339/ using document root <none> and password file <none>
I20260807 11:21:34.586431 22536 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:34.586607 22536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:34.586912 22536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:34.590087 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "b5cc83c799e84be984028f82a742571a"
format_stamp: "Formatted at 2026-08-07 11:21:34 on dist-test-slave-5l3k"
server_key: "893149bf172d266013f4472b0fc30207"
server_key_iv: "71e32c179909e00f16b0c1ea36f8cefb"
server_key_version: "encryptionkey@0"
I20260807 11:21:34.591012 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "b5cc83c799e84be984028f82a742571a"
format_stamp: "Formatted at 2026-08-07 11:21:34 on dist-test-slave-5l3k"
server_key: "893149bf172d266013f4472b0fc30207"
server_key_iv: "71e32c179909e00f16b0c1ea36f8cefb"
server_key_version: "encryptionkey@0"
I20260807 11:21:34.596796 22536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260807 11:21:34.600778 22553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.602360 22536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:21:34.602587 22536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "b5cc83c799e84be984028f82a742571a"
format_stamp: "Formatted at 2026-08-07 11:21:34 on dist-test-slave-5l3k"
server_key: "893149bf172d266013f4472b0fc30207"
server_key_iv: "71e32c179909e00f16b0c1ea36f8cefb"
server_key_version: "encryptionkey@0"
I20260807 11:21:34.602825 22536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:34.624655 22536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:34.629340 22536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:34.629668 22536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:34.630793 22536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:34.632398 22536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:34.632521 22536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.632692 22536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:34.632769 22536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:34.669917 22536 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:35197
I20260807 11:21:34.669973 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:35197 every 8 connection(s)
I20260807 11:21:34.671516 22536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:34.681093 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22536
I20260807 11:21:34.681314 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:34.681833 2942 external_mini_cluster.cc:1428] Setting key a31b63953d070c4a39de6d0125e9282d
Aug 07 11:21:34 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:34.693961 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:34.674359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:49327 (local address 127.2.223.190:35935)
0807 11:21:34.674881 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:34.674886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:34.676016 (+ 1130us) server_negotiation.cc:407] Connection header received
0807 11:21:34.678090 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:34.678096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:34.678207 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:34.678337 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:34.680095 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.680848 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.682823 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.683343 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.683502 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:34.687469 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:34.687474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:34.687478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:34.687519 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:34.690309 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:34.691291 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:34.691296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:34.691298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:34.691452 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:34.691983 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:34.691987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:34.691989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:34.692307 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:34.692482 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:34.693304 (+ 822us) server_negotiation.cc:299] Negotiation successful
0807 11:21:34.693571 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260807 11:21:34.695175 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:34.675043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35935 (local address 127.2.223.130:49327)
0807 11:21:34.675825 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:34.675873 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:34.677764 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:34.678654 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:34.678664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:34.679250 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:34.679884 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:34.679903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.681034 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.681038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:34.682655 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:34.682671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:34.683629 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:34.683639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:34.683892 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:34.684643 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:34.684675 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:34.687137 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:34.690582 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:34.690591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:34.690614 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:34.691123 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:34.691584 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:34.691589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:34.691593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:34.691833 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:34.692455 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:34.692464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:34.692881 (+ 417us) client_negotiation.cc:769] Sending connection context
0807 11:21:34.693178 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 11:21:34.693815 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":306,"threads_started":1}
I20260807 11:21:34.696874 22671 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35935
I20260807 11:21:34.697242 22671 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:34.698057 22671 heartbeater.cc:511] Master 127.2.223.190:35935 requested a full tablet report, sending...
I20260807 11:21:34.699945 22328 ts_manager.cc:194] Registered new tserver with Master: b5cc83c799e84be984028f82a742571a (127.2.223.130:35197)
I20260807 11:21:34.700883 22328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:49327
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:21:34.765775 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:21:35.115954 22530 heartbeater.cc:503] Master 127.2.223.190:35935 was elected leader, sending a full tablet report...
W20260807 11:21:35.148108 22677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:35.148437 22677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:35.148537 22677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:35.156204 22677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:35.156342 22677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:35.156462 22677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:35.167018 22677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:35.168437 22677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:35.169869 22677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:35.184919 22682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:35.184929 22685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:35.184885 22683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:35.186259 22677 server_base.cc:1034] running on GCE node
I20260807 11:21:35.187021 22677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:35.188180 22677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:35.189409 22677 hybrid_clock.cc:648] HybridClock initialized: now 1786101695189331 us; error 72 us; skew 500 ppm
I20260807 11:21:35.189791 22677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:35 dist-test-slave-5l3k krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:35.193658 22677 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:21:35.195559 22677 webserver.cc:460] Webserver started at http://127.2.223.131:38955/ using document root <none> and password file <none>
I20260807 11:21:35.196380 22677 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:35.196532 22677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:35.196900 22677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:35.200325 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "c4620b59e8d74d2690ca32136195ee04"
format_stamp: "Formatted at 2026-08-07 11:21:35 on dist-test-slave-5l3k"
server_key: "77c0c68677374ed56e4b630f0c7c2bb4"
server_key_iv: "f0e9e44b91d48e819c68a788439fe09b"
server_key_version: "encryptionkey@0"
I20260807 11:21:35.201298 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "c4620b59e8d74d2690ca32136195ee04"
format_stamp: "Formatted at 2026-08-07 11:21:35 on dist-test-slave-5l3k"
server_key: "77c0c68677374ed56e4b630f0c7c2bb4"
server_key_iv: "f0e9e44b91d48e819c68a788439fe09b"
server_key_version: "encryptionkey@0"
I20260807 11:21:35.207935 22677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 11:21:35.212328 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:35.214064 22677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:21:35.214298 22677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "c4620b59e8d74d2690ca32136195ee04"
format_stamp: "Formatted at 2026-08-07 11:21:35 on dist-test-slave-5l3k"
server_key: "77c0c68677374ed56e4b630f0c7c2bb4"
server_key_iv: "f0e9e44b91d48e819c68a788439fe09b"
server_key_version: "encryptionkey@0"
I20260807 11:21:35.214566 22677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:35.246958 22677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:35.251837 22677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:35.252195 22677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:35.253367 22677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:35.254967 22677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:35.255084 22677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:35.255268 22677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:35.255354 22677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:35.294138 22677 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:35291
I20260807 11:21:35.294183 22809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:35291 every 8 connection(s)
I20260807 11:21:35.295797 22677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:35.305884 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22677
I20260807 11:21:35.306110 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:35.306730 2942 external_mini_cluster.cc:1428] Setting key 5deaecac5d1d64ff446149252656019e
Aug 07 11:21:35 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101695, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:35.321098 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.298977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:39157 (local address 127.2.223.190:35935)
0807 11:21:35.299522 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.299528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.300784 (+ 1256us) server_negotiation.cc:407] Connection header received
0807 11:21:35.302930 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.302936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.303066 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.303199 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:35.305233 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.306606 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.308954 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.309579 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.309717 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.314044 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:35.314050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:35.314053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:35.314093 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:35.317019 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.318011 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.318016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.318019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.318179 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.318831 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.318835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.318838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.319201 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:35.319354 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.320426 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.320716 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
I20260807 11:21:35.322396 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.299712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35935 (local address 127.2.223.131:39157)
0807 11:21:35.300577 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.300635 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.302592 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.303482 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.303492 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.304276 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:35.304973 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.304995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.306858 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.306866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.308777 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.308795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.309879 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.309893 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.310191 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.310977 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:35.311002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:35.313804 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:35.317230 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:35.317239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:35.317256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:35.317817 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:35.318358 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:35.318362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:35.318364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:35.318675 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:35.319411 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:35.319419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:35.319895 (+ 476us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.320208 (+ 313us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.320741 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":397,"threads_started":1}
I20260807 11:21:35.324035 22810 heartbeater.cc:347] Connected to a master server at 127.2.223.190:35935
I20260807 11:21:35.324379 22810 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:35.325203 22810 heartbeater.cc:511] Master 127.2.223.190:35935 requested a full tablet report, sending...
I20260807 11:21:35.326844 22328 ts_manager.cc:194] Registered new tserver with Master: c4620b59e8d74d2690ca32136195ee04 (127.2.223.131:35291)
I20260807 11:21:35.327747 22328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:39157
I20260807 11:21:35.336002 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:35.357976 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.340755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.2.223.190:35935)
0807 11:21:35.340942 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.340947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.341426 (+ 479us) server_negotiation.cc:407] Connection header received
0807 11:21:35.341584 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.341588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.341681 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.341785 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:35.343595 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.344431 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.345928 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.346392 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.346698 (+ 306us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.349889 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:35.349905 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:35.349909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:35.349955 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:35.354111 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.355369 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.355376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.355380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.355570 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.356112 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.356118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.356122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.356628 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:35.356922 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.357395 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.357625 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:21:35.364130 22328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:35.371794 22328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:35.400049 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.388526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:35197 (local address 127.0.0.1:38482)
0807 11:21:35.389359 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.389380 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.389522 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.391117 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.391132 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.391174 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:35.391699 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.391714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.394143 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.394149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.396684 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.396707 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.398939 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.398946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.399183 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.399252 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.399389 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.399622 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":494,"threads_started":1}
I20260807 11:21:35.401641 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.388897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.2.223.130:35197)
0807 11:21:35.390339 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.390370 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.390416 (+ 46us) server_negotiation.cc:407] Connection header received
0807 11:21:35.390552 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.390560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.390844 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.391972 (+ 1128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:35.392279 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.393957 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.396958 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.398654 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.400692 (+ 2038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.400969 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.401059 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.401277 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":895,"threads_started":1}
I20260807 11:21:35.403738 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.386797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42881 (local address 127.0.0.1:44112)
0807 11:21:35.387995 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.388040 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.388223 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.392048 (+ 3825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.392060 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.392093 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:35.392575 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.392600 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.396110 (+ 3510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.396116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.398332 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.398350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.402671 (+ 4321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.402680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.402958 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.403027 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.403161 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.403314 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":766,"threads_started":1}
I20260807 11:21:35.403951 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.387206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.2.223.129:42881)
0807 11:21:35.389009 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.389022 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.389065 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:21:35.389195 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.389206 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.389534 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.389798 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:35.392818 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.394382 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.398627 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.402352 (+ 3725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.402569 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.402837 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.403258 (+ 421us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.403470 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1579,"thread_start_us":1405,"threads_started":1}
I20260807 11:21:35.408135 22461 tablet_service.cc:1467] Processing CreateTablet for tablet 8657e61091d24b6da837f4f96da04625 (DEFAULT_TABLE table=test-table [id=79f645442ae347efa8374f834261e9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:35.407981 22603 tablet_service.cc:1467] Processing CreateTablet for tablet 8657e61091d24b6da837f4f96da04625 (DEFAULT_TABLE table=test-table [id=79f645442ae347efa8374f834261e9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:35.409565 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8657e61091d24b6da837f4f96da04625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:35.409614 22603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8657e61091d24b6da837f4f96da04625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:35.411039 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.390166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:35291 (local address 127.0.0.1:44198)
0807 11:21:35.400351 (+ 10185us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.400376 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.400496 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.403290 (+ 2794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.403295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.403330 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:35.403528 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.403541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.405992 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.405998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.407995 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.408027 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.410061 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.410069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.410288 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.410332 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.410452 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.410655 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10073,"thread_start_us":12956,"threads_started":1}
I20260807 11:21:35.411149 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.389979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.2.223.131:35291)
0807 11:21:35.402512 (+ 12533us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.402523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.402563 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:21:35.402687 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.402696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.402999 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.403745 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:35.404094 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.405778 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.408254 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.409838 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.410001 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.410246 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.410566 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.410753 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12286,"thread_start_us":12102,"threads_started":1}
I20260807 11:21:35.419950 22743 tablet_service.cc:1467] Processing CreateTablet for tablet 8657e61091d24b6da837f4f96da04625 (DEFAULT_TABLE table=test-table [id=79f645442ae347efa8374f834261e9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:35.422144 22743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8657e61091d24b6da837f4f96da04625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:35.424947 22832 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Bootstrap starting.
I20260807 11:21:35.427161 22831 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Bootstrap starting.
I20260807 11:21:35.429633 22832 tablet_bootstrap.cc:654] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:35.431097 22832 log.cc:824] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:35.431245 22831 tablet_bootstrap.cc:654] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:35.432613 22831 log.cc:824] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:35.434664 22832 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: No bootstrap required, opened a new log
I20260807 11:21:35.434998 22832 ts_tablet_manager.cc:1397] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 11:21:35.436173 22831 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: No bootstrap required, opened a new log
I20260807 11:21:35.436471 22831 ts_tablet_manager.cc:1397] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 11:21:35.439172 22835 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Bootstrap starting.
I20260807 11:21:35.444011 22835 tablet_bootstrap.cc:654] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:35.445535 22835 log.cc:824] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:35.445441 22831 raft_consensus.cc:348] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.445936 22831 raft_consensus.cc:374] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:35.446059 22831 raft_consensus.cc:729] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a25bef0cde43239ea31c19701eac21, State: Initialized, Role: FOLLOWER
I20260807 11:21:35.445883 22832 raft_consensus.cc:348] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.446401 22832 raft_consensus.cc:374] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:35.446532 22832 raft_consensus.cc:729] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cc83c799e84be984028f82a742571a, State: Initialized, Role: FOLLOWER
I20260807 11:21:35.446954 22831 consensus_queue.cc:260] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.447188 22832 consensus_queue.cc:260] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.449038 22831 ts_tablet_manager.cc:1428] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260807 11:21:35.449882 22835 tablet_bootstrap.cc:492] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: No bootstrap required, opened a new log
I20260807 11:21:35.450169 22671 heartbeater.cc:503] Master 127.2.223.190:35935 was elected leader, sending a full tablet report...
I20260807 11:21:35.450210 22835 ts_tablet_manager.cc:1397] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260807 11:21:35.450976 22832 ts_tablet_manager.cc:1428] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260807 11:21:35.459839 22835 raft_consensus.cc:348] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.460359 22835 raft_consensus.cc:374] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:35.460492 22835 raft_consensus.cc:729] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4620b59e8d74d2690ca32136195ee04, State: Initialized, Role: FOLLOWER
I20260807 11:21:35.461174 22835 consensus_queue.cc:260] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.462574 22810 heartbeater.cc:503] Master 127.2.223.190:35935 was elected leader, sending a full tablet report...
I20260807 11:21:35.463025 22835 ts_tablet_manager.cc:1428] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260807 11:21:35.503738 22839 raft_consensus.cc:482] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:35.504068 22839 raft_consensus.cc:504] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.505929 22839 leader_election.cc:288] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43a25bef0cde43239ea31c19701eac21 (127.2.223.129:42881), c4620b59e8d74d2690ca32136195ee04 (127.2.223.131:35291)
I20260807 11:21:35.515877 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.507752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:32991 (local address 127.2.223.131:35291)
0807 11:21:35.507989 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.507997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.508278 (+ 281us) server_negotiation.cc:407] Connection header received
0807 11:21:35.508471 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.508475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.508534 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.508613 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:35.509167 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.510232 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.513233 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.514688 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.514895 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.514984 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.515481 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.515631 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:21:35.515959 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.507451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:35291 (local address 127.2.223.130:32991)
0807 11:21:35.508157 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.508177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.508333 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.508661 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.508665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.508697 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:35.508995 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.509003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.510375 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.510380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.512971 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.513002 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.515011 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.515022 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.515222 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.515276 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.515394 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.515517 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":49,"thread_start_us":373,"threads_started":1}
I20260807 11:21:35.517077 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.506566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:51571 (local address 127.2.223.129:42881)
0807 11:21:35.506887 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.506896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.507478 (+ 582us) server_negotiation.cc:407] Connection header received
0807 11:21:35.507705 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.507711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.507818 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.507944 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:35.508814 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.509924 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.513233 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.514695 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.514895 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.514995 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.516722 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.516881 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:21:35.517268 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.506378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42881 (local address 127.2.223.130:51571)
0807 11:21:35.507301 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:35.507325 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:35.507499 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:35.508110 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:35.508115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:35.508150 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:35.508636 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.508649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.510122 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.510126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.513000 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:35.513010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.516304 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.516312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:35.516506 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.516539 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:21:35.516650 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:21:35.516849 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":190,"thread_start_us":529,"threads_started":1}
I20260807 11:21:35.517439 22763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8657e61091d24b6da837f4f96da04625" candidate_uuid: "b5cc83c799e84be984028f82a742571a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4620b59e8d74d2690ca32136195ee04" is_pre_election: true
I20260807 11:21:35.518229 22763 raft_consensus.cc:2454] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5cc83c799e84be984028f82a742571a in term 0.
I20260807 11:21:35.518406 22483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8657e61091d24b6da837f4f96da04625" candidate_uuid: "b5cc83c799e84be984028f82a742571a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a25bef0cde43239ea31c19701eac21" is_pre_election: true
I20260807 11:21:35.519062 22483 raft_consensus.cc:2454] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5cc83c799e84be984028f82a742571a in term 0.
I20260807 11:21:35.519120 22557 leader_election.cc:302] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5cc83c799e84be984028f82a742571a, c4620b59e8d74d2690ca32136195ee04; no voters:
I20260807 11:21:35.519665 22839 raft_consensus.cc:2781] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:35.519832 22839 raft_consensus.cc:482] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:35.519979 22839 raft_consensus.cc:3037] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:35.522377 22839 raft_consensus.cc:504] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.523123 22839 leader_election.cc:288] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [CANDIDATE]: Term 1 election: Requested vote from peers 43a25bef0cde43239ea31c19701eac21 (127.2.223.129:42881), c4620b59e8d74d2690ca32136195ee04 (127.2.223.131:35291)
I20260807 11:21:35.523785 22483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8657e61091d24b6da837f4f96da04625" candidate_uuid: "b5cc83c799e84be984028f82a742571a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a25bef0cde43239ea31c19701eac21"
I20260807 11:21:35.523866 22763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8657e61091d24b6da837f4f96da04625" candidate_uuid: "b5cc83c799e84be984028f82a742571a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4620b59e8d74d2690ca32136195ee04"
I20260807 11:21:35.524006 22483 raft_consensus.cc:3037] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:35.524144 22763 raft_consensus.cc:3037] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:35.526458 22483 raft_consensus.cc:2454] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5cc83c799e84be984028f82a742571a in term 1.
I20260807 11:21:35.526691 22763 raft_consensus.cc:2454] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5cc83c799e84be984028f82a742571a in term 1.
I20260807 11:21:35.527105 22557 leader_election.cc:302] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a25bef0cde43239ea31c19701eac21, b5cc83c799e84be984028f82a742571a; no voters:
I20260807 11:21:35.527468 22839 raft_consensus.cc:2781] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:35.527972 22839 raft_consensus.cc:686] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 1 LEADER]: Becoming Leader. State: Replica: b5cc83c799e84be984028f82a742571a, State: Running, Role: LEADER
I20260807 11:21:35.528492 22839 consensus_queue.cc:237] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } }
I20260807 11:21:35.533921 22327 catalog_manager.cc:5495] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a reported cstate change: term changed from 0 to 1, leader changed from <none> to b5cc83c799e84be984028f82a742571a (127.2.223.130). New cstate: current_term: 1 leader_uuid: "b5cc83c799e84be984028f82a742571a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5cc83c799e84be984028f82a742571a" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 35197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 } health_report { overall_health: UNKNOWN } } }
W20260807 11:21:35.547915 22811 tablet.cc:2369] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:35.560850 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.554940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.2.223.130:35197)
0807 11:21:35.555207 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.555212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.555232 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:21:35.555400 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.555404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.555498 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.555638 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:35.556103 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.556834 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.558318 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.558803 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.558913 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.559522 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:35.560081 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:35.560217 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.560415 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.560505 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:21:35.570107 22763 raft_consensus.cc:1261] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 1 FOLLOWER]: Refusing update from remote peer b5cc83c799e84be984028f82a742571a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:35.570155 22483 raft_consensus.cc:1261] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 1 FOLLOWER]: Refusing update from remote peer b5cc83c799e84be984028f82a742571a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:35.571331 22843 consensus_queue.cc:1035] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4620b59e8d74d2690ca32136195ee04" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:35.571717 22839 consensus_queue.cc:1035] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a25bef0cde43239ea31c19701eac21" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:21:35.577414 22531 tablet.cc:2369] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:35.586880 22849 mvcc.cc:205] Tried to move back new op lower bound from 7315872545044156416 to 7315872544889044992. Current Snapshot: MvccSnapshot[applied={T|T < 7315872545044156416}]
I20260807 11:21:35.603152 22327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:35.603484 22327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:35.607970 22327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b39dd1abcb2e499ab51180344a33e90c: Sending DeleteTablet for 3 replicas of tablet 8657e61091d24b6da837f4f96da04625
I20260807 11:21:35.609589 22743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8657e61091d24b6da837f4f96da04625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:35 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:44198
I20260807 11:21:35.609570 22461 tablet_service.cc:1514] Processing DeleteTablet for tablet 8657e61091d24b6da837f4f96da04625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:35 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:44112
I20260807 11:21:35.610814 22603 tablet_service.cc:1514] Processing DeleteTablet for tablet 8657e61091d24b6da837f4f96da04625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:35 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:38482
I20260807 11:21:35.614743 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22392
I20260807 11:21:35.620893 22859 tablet_replica.cc:330] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: stopping tablet replica
I20260807 11:21:35.621620 22859 raft_consensus.cc:2229] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:35.622659 22859 raft_consensus.cc:2258] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:35.628554 22859 ts_tablet_manager.cc:1905] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:35.629459 22857 tablet_replica.cc:330] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: stopping tablet replica
I20260807 11:21:35.629966 22857 raft_consensus.cc:2229] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:35.630578 22857 raft_consensus.cc:2258] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:35.634447 22857 ts_tablet_manager.cc:1905] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:35.637926 22858 tablet_replica.cc:330] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: stopping tablet replica
I20260807 11:21:35.641085 22859 ts_tablet_manager.cc:1918] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:35.641314 22859 log.cc:1196] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/8657e61091d24b6da837f4f96da04625
Aug 07 11:21:35 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:21:35.641971 22859 ts_tablet_manager.cc:1939] T 8657e61091d24b6da837f4f96da04625 P b5cc83c799e84be984028f82a742571a: Deleting consensus metadata
I20260807 11:21:35.642325 22858 raft_consensus.cc:2229] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:35.643043 22858 raft_consensus.cc:2258] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:35.643117 22857 ts_tablet_manager.cc:1918] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:35.643326 22857 log.cc:1196] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/8657e61091d24b6da837f4f96da04625
I20260807 11:21:35.644233 22857 ts_tablet_manager.cc:1939] T 8657e61091d24b6da837f4f96da04625 P 43a25bef0cde43239ea31c19701eac21: Deleting consensus metadata
I20260807 11:21:35.644263 22314 catalog_manager.cc:4841] TS b5cc83c799e84be984028f82a742571a (127.2.223.130:35197): tablet 8657e61091d24b6da837f4f96da04625 (table test-table [id=79f645442ae347efa8374f834261e9df]) successfully deleted
I20260807 11:21:35.646029 22314 catalog_manager.cc:4841] TS 43a25bef0cde43239ea31c19701eac21 (127.2.223.129:42881): tablet 8657e61091d24b6da837f4f96da04625 (table test-table [id=79f645442ae347efa8374f834261e9df]) successfully deleted
I20260807 11:21:35.646531 22858 ts_tablet_manager.cc:1905] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:35.653949 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.616176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.2.223.129:42881)
0807 11:21:35.616428 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.616438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.625895 (+ 9457us) server_negotiation.cc:407] Connection header received
0807 11:21:35.626617 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.626627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.626740 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.626834 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:35.630686 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.631911 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.634551 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.635281 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.635420 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.647203 (+ 11783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:35.647217 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:35.647232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:35.647274 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:35.650448 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.651436 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.651445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.651447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.651606 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.652077 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.652082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.652084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.652496 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:35.652853 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.653248 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.653505 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:21:35.654544 22858 ts_tablet_manager.cc:1918] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:35.654795 22858 log.cc:1196] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/8657e61091d24b6da837f4f96da04625
I20260807 11:21:35.654791 22525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:35.655412 22858 ts_tablet_manager.cc:1939] T 8657e61091d24b6da837f4f96da04625 P c4620b59e8d74d2690ca32136195ee04: Deleting consensus metadata
I20260807 11:21:35.657792 22314 catalog_manager.cc:4841] TS c4620b59e8d74d2690ca32136195ee04 (127.2.223.131:35291): tablet 8657e61091d24b6da837f4f96da04625 (table test-table [id=79f645442ae347efa8374f834261e9df]) successfully deleted
I20260807 11:21:35.769129 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22392
I20260807 11:21:35.793401 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22536
Aug 07 11:21:35 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:21:35.812479 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.794310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.2.223.130:35197)
0807 11:21:35.794529 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.794536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.794645 (+ 109us) server_negotiation.cc:407] Connection header received
0807 11:21:35.794863 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.794869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.794966 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.795085 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:35.796462 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.797374 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.798964 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.799611 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.799782 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.805392 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:35.805408 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:35.805425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:35.805476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:35.809085 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.810070 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.810078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.810080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.810244 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.810748 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.810752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.810754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.811152 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:35.811542 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.811796 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.812039 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:21:35.813287 22666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:35.936628 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22536
I20260807 11:21:35.961946 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22677
Aug 07 11:21:35 dist-test-slave-5l3k krb5kdc[22280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:21:35.979501 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:35.962812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.2.223.131:35291)
0807 11:21:35.963058 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:35.963065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:35.963092 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:21:35.963206 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:35.963210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:35.963285 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:35.963436 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:35.964825 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.965611 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.967150 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:35.967768 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:35.967908 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:35.972874 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:35.972894 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:35.972914 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:35.972961 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:35.976051 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.976975 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.976983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.976986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.977162 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:35.977669 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:35.977674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:35.977676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:35.978144 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:35.978412 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:35.978885 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 11:21:35.979103 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:21:35.980309 22805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:36.074607 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22677
I20260807 11:21:36.097054 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22293
I20260807 11:21:36.098279 22360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:36.213925 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22293
2026-08-07T11:21:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4974 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T11:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:36Z Disabled control of system clock
I20260807 11:21:36.325152 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41015
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:36.711627 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:36.711969 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:36.712067 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:36.719724 22869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:36.719862 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:36.719901 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:36.719961 22869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:36.720016 22869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:36.729698 22869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:36.731285 22869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:36.733388 22869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:36.747093 22874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:36.747916 22877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:36.748080 22869 server_base.cc:1034] running on GCE node
W20260807 11:21:36.748261 22875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:36.749418 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:36.751325 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:36.752635 22869 hybrid_clock.cc:648] HybridClock initialized: now 1786101696752575 us; error 75 us; skew 500 ppm
I20260807 11:21:36.753104 22869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:36.755923 22869 webserver.cc:460] Webserver started at http://127.2.223.190:37619/ using document root <none> and password file <none>
I20260807 11:21:36.756784 22869 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:36.756944 22869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:36.757320 22869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:36.760838 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "4c86dec0e048405fa08c4aa5661dfc1d"
format_stamp: "Formatted at 2026-08-07 11:21:36 on dist-test-slave-5l3k"
server_key: "90d7438f7f15d45ab55772427f905bea"
server_key_iv: "11e872cf75357d3b2e7fd0fe8ec0e7f0"
server_key_version: "encryptionkey@0"
I20260807 11:21:36.761821 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "4c86dec0e048405fa08c4aa5661dfc1d"
format_stamp: "Formatted at 2026-08-07 11:21:36 on dist-test-slave-5l3k"
server_key: "90d7438f7f15d45ab55772427f905bea"
server_key_iv: "11e872cf75357d3b2e7fd0fe8ec0e7f0"
server_key_version: "encryptionkey@0"
I20260807 11:21:36.768316 22869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260807 11:21:36.772795 22884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:36.774467 22869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:21:36.774767 22869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "4c86dec0e048405fa08c4aa5661dfc1d"
format_stamp: "Formatted at 2026-08-07 11:21:36 on dist-test-slave-5l3k"
server_key: "90d7438f7f15d45ab55772427f905bea"
server_key_iv: "11e872cf75357d3b2e7fd0fe8ec0e7f0"
server_key_version: "encryptionkey@0"
I20260807 11:21:36.775033 22869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:36.808717 22869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:36.810074 22869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:36.810411 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:36.831187 22869 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41015
I20260807 11:21:36.831219 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41015 every 8 connection(s)
I20260807 11:21:36.832748 22869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:36.835332 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22869
I20260807 11:21:36.835484 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:36.835976 2942 external_mini_cluster.cc:1428] Setting key bafd69a5553ffe709f7d586855ba71c0
I20260807 11:21:36.837361 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:36.852831 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:36.838124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.2.223.190:41015)
0807 11:21:36.839303 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:36.839314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:36.839361 (+ 47us) server_negotiation.cc:407] Connection header received
0807 11:21:36.841121 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:36.841148 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:36.841725 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:36.842185 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:36.842849 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:36.844254 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:36.845798 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:36.846621 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:36.846851 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:36.848311 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:36.848326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:36.848354 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:36.849519 (+ 1165us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:36.849681 (+ 162us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:36.849733 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:36.849911 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:36.850399 (+ 488us) server_negotiation.cc:299] Negotiation successful
0807 11:21:36.851065 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":490,"threads_started":1}
I20260807 11:21:36.853120 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: Bootstrap starting.
I20260807 11:21:36.857690 22939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:36.858948 22939 log.cc:824] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:36.862150 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: No bootstrap required, opened a new log
I20260807 11:21:36.870762 22939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } }
I20260807 11:21:36.871223 22939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:36.871313 22939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c86dec0e048405fa08c4aa5661dfc1d, State: Initialized, Role: FOLLOWER
I20260807 11:21:36.871897 22939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } }
I20260807 11:21:36.872124 22939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:36.872239 22939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:36.872380 22939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:36.875146 22939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } }
I20260807 11:21:36.875888 22939 leader_election.cc:302] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c86dec0e048405fa08c4aa5661dfc1d; no voters:
I20260807 11:21:36.876508 22939 leader_election.cc:288] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:36.876708 22945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:36.877584 22945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [term 1 LEADER]: Becoming Leader. State: Replica: 4c86dec0e048405fa08c4aa5661dfc1d, State: Running, Role: LEADER
I20260807 11:21:36.878280 22945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } }
I20260807 11:21:36.879097 22939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:36.881713 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c86dec0e048405fa08c4aa5661dfc1d. Latest consensus state: current_term: 1 leader_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } } }
I20260807 11:21:36.882159 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:36.882937 22946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86dec0e048405fa08c4aa5661dfc1d" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41015 } } }
I20260807 11:21:36.883286 22946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:36.883503 22950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:36.890555 22950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:36.901950 22950 catalog_manager.cc:1347] Generated new cluster ID: 19be8cb700744a1aa3c49ef8836ea249
I20260807 11:21:36.902127 22950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:36.934931 22950 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:36.936360 22950 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:36.948493 22950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: Generated new TSK 0
I20260807 11:21:36.949771 22950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:36.960989 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--builtin_ntp_servers=127.2.223.148:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:37.342875 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:37.343204 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:37.343370 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:37.351033 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:37.351217 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:37.351413 22964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:37.361248 22964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:37.363013 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:37.365136 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:37.379953 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:37.379953 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:37.379997 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:37.381726 22964 server_base.cc:1034] running on GCE node
I20260807 11:21:37.382562 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:37.383947 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:37.385205 22964 hybrid_clock.cc:648] HybridClock initialized: now 1786101697385150 us; error 58 us; skew 500 ppm
I20260807 11:21:37.385535 22964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:37.388221 22964 webserver.cc:460] Webserver started at http://127.2.223.129:38043/ using document root <none> and password file <none>
I20260807 11:21:37.388976 22964 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:37.389076 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:37.389468 22964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:37.392714 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "a83b13e8eeaf4e4789222ec48d486eed"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "624f4030405cde415e0746a0231cc2be"
server_key_iv: "a53f159253796e77d5dd5f217932dabb"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.393548 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "a83b13e8eeaf4e4789222ec48d486eed"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "624f4030405cde415e0746a0231cc2be"
server_key_iv: "a53f159253796e77d5dd5f217932dabb"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.399681 22964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 11:21:37.403635 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:37.405334 22964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 11:21:37.405571 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "a83b13e8eeaf4e4789222ec48d486eed"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "624f4030405cde415e0746a0231cc2be"
server_key_iv: "a53f159253796e77d5dd5f217932dabb"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.405912 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:37.432179 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:37.433063 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:37.433357 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:37.434396 22964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:37.436021 22964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:37.436117 22964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:37.436196 22964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:37.436254 22964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:37.472681 22964 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:42575
I20260807 11:21:37.472788 23096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:42575 every 8 connection(s)
I20260807 11:21:37.474237 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:37.481743 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 22964
I20260807 11:21:37.481925 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:37.482426 2942 external_mini_cluster.cc:1428] Setting key 48656a1a6a76f46b742d6c8a0936e894
I20260807 11:21:37.485860 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--builtin_ntp_servers=127.2.223.148:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:37.489677 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:37.477091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:59567 (local address 127.2.223.190:41015)
0807 11:21:37.477652 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:37.477660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:37.479189 (+ 1529us) server_negotiation.cc:407] Connection header received
0807 11:21:37.481279 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:37.481285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:37.481391 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:37.481541 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:37.482913 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:37.483881 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:37.486023 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:37.486818 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:37.486986 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:37.488597 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:37.488608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:37.488615 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:37.488655 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:37.488715 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:37.488736 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:37.488837 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:37.489068 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 11:21:37.489196 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
I20260807 11:21:37.490757 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:37.477787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41015 (local address 127.2.223.129:59567)
0807 11:21:37.478971 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:37.479013 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:37.480944 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:37.481761 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:37.481771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:37.481918 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:37.482714 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:37.482737 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:37.484095 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:37.484102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:37.485811 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:37.485827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:37.487146 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:37.487154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:37.487391 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:37.488209 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:37.488212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:37.488257 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:37.488351 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:37.488843 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:37.488853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:37.488860 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:21:37.489021 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:21:37.489400 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260807 11:21:37.492681 23097 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41015
I20260807 11:21:37.493104 23097 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:37.494226 23097 heartbeater.cc:511] Master 127.2.223.190:41015 requested a full tablet report, sending...
I20260807 11:21:37.496877 22902 ts_manager.cc:194] Registered new tserver with Master: a83b13e8eeaf4e4789222ec48d486eed (127.2.223.129:42575)
I20260807 11:21:37.498642 22902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.129:59567
W20260807 11:21:37.879962 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:37.880375 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:37.880492 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:37.889225 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:37.889359 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:37.889483 23101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:37.900060 23101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:37.901420 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:37.903458 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:37.918035 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:37.918053 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:37.918660 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:37.918921 23101 server_base.cc:1034] running on GCE node
I20260807 11:21:37.919605 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:37.920687 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:37.921945 23101 hybrid_clock.cc:648] HybridClock initialized: now 1786101697921895 us; error 43 us; skew 500 ppm
I20260807 11:21:37.922319 23101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:37.924940 23101 webserver.cc:460] Webserver started at http://127.2.223.130:38749/ using document root <none> and password file <none>
I20260807 11:21:37.925736 23101 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:37.925887 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:37.926256 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:37.929455 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "a483363da1584710b3f495d177e2f423"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "eac53da75c14fd7860422a3802d9614e"
server_key_iv: "4fa0cedc7a697d373d472657f0dd4552"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.930315 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "a483363da1584710b3f495d177e2f423"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "eac53da75c14fd7860422a3802d9614e"
server_key_iv: "4fa0cedc7a697d373d472657f0dd4552"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.935791 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:21:37.939958 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:37.941373 23101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:21:37.941567 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "a483363da1584710b3f495d177e2f423"
format_stamp: "Formatted at 2026-08-07 11:21:37 on dist-test-slave-5l3k"
server_key: "eac53da75c14fd7860422a3802d9614e"
server_key_iv: "4fa0cedc7a697d373d472657f0dd4552"
server_key_version: "encryptionkey@0"
I20260807 11:21:37.941816 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:37.963642 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:37.964627 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:37.964977 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:37.966079 23101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:37.967775 23101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:37.967912 23101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:37.968065 23101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:37.968158 23101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:38.004918 23101 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:46179
I20260807 11:21:38.005023 23234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:46179 every 8 connection(s)
I20260807 11:21:38.006340 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:38.010210 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23101
I20260807 11:21:38.010389 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:38.010973 2942 external_mini_cluster.cc:1428] Setting key c0ef178d763ed7524a68001228f34b64
I20260807 11:21:38.013895 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--builtin_ntp_servers=127.2.223.148:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:38.020223 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.009130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:41167 (local address 127.2.223.190:41015)
0807 11:21:38.009684 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.009692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.011237 (+ 1545us) server_negotiation.cc:407] Connection header received
0807 11:21:38.013357 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.013362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.013452 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.013576 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:38.014838 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.015612 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.017235 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.017651 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.017806 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.019300 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:38.019305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:38.019308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:38.019328 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:38.019373 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:38.019385 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:38.019486 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.019729 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.019838 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":240,"threads_started":1}
I20260807 11:21:38.021322 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.009847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41015 (local address 127.2.223.130:41167)
0807 11:21:38.011046 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.011093 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.013080 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.013828 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.013836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.013930 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:38.014659 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.014679 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.015791 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.015795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.017068 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.017081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.017906 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.017913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.018163 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.018950 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:38.018952 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:38.018994 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:38.019084 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:38.019487 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:38.019502 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:38.019512 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.019710 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.020136 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":369,"threads_started":1}
I20260807 11:21:38.022922 23235 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41015
I20260807 11:21:38.023265 23235 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:38.024068 23235 heartbeater.cc:511] Master 127.2.223.190:41015 requested a full tablet report, sending...
I20260807 11:21:38.025723 22902 ts_manager.cc:194] Registered new tserver with Master: a483363da1584710b3f495d177e2f423 (127.2.223.130:46179)
I20260807 11:21:38.026991 22902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.130:41167
W20260807 11:21:38.388681 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:38.389015 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:38.389122 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:38.396976 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:38.397126 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:38.397245 23239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:38.407222 23239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:38.408599 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:38.410712 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:38.425774 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:38.425836 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:38.425813 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:38.427254 23239 server_base.cc:1034] running on GCE node
I20260807 11:21:38.428063 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:38.429176 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:38.430451 23239 hybrid_clock.cc:648] HybridClock initialized: now 1786101698430395 us; error 52 us; skew 500 ppm
I20260807 11:21:38.430871 23239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:38.436769 23239 webserver.cc:460] Webserver started at http://127.2.223.131:34599/ using document root <none> and password file <none>
I20260807 11:21:38.437603 23239 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:38.437778 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:38.438158 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:38.441327 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "49eb39376bed4898845c028406ff2fdc"
format_stamp: "Formatted at 2026-08-07 11:21:38 on dist-test-slave-5l3k"
server_key: "fb6448ed027d0623de6cee700e18ae29"
server_key_iv: "d1e36955e3e50062b4adcc48fd2c8699"
server_key_version: "encryptionkey@0"
I20260807 11:21:38.442209 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "49eb39376bed4898845c028406ff2fdc"
format_stamp: "Formatted at 2026-08-07 11:21:38 on dist-test-slave-5l3k"
server_key: "fb6448ed027d0623de6cee700e18ae29"
server_key_iv: "d1e36955e3e50062b4adcc48fd2c8699"
server_key_version: "encryptionkey@0"
I20260807 11:21:38.448652 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:38.452967 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:38.454684 23239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:21:38.454921 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "49eb39376bed4898845c028406ff2fdc"
format_stamp: "Formatted at 2026-08-07 11:21:38 on dist-test-slave-5l3k"
server_key: "fb6448ed027d0623de6cee700e18ae29"
server_key_iv: "d1e36955e3e50062b4adcc48fd2c8699"
server_key_version: "encryptionkey@0"
I20260807 11:21:38.455212 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:38.470022 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:38.471107 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:38.471489 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:38.472864 23239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:38.475080 23239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:38.475217 23239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:38.475340 23239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:38.475420 23239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:38.503250 23097 heartbeater.cc:503] Master 127.2.223.190:41015 was elected leader, sending a full tablet report...
I20260807 11:21:38.513602 23239 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:42223
I20260807 11:21:38.513659 23372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:42223 every 8 connection(s)
I20260807 11:21:38.515118 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:38.523418 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23239
I20260807 11:21:38.523615 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:38.524134 2942 external_mini_cluster.cc:1428] Setting key d14e62c728572c09f446c45a24328403
I20260807 11:21:38.529628 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.518158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:57017 (local address 127.2.223.190:41015)
0807 11:21:38.518406 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.518411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.519961 (+ 1550us) server_negotiation.cc:407] Connection header received
0807 11:21:38.521962 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.521968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.522077 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.522200 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:38.523775 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.524616 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.526472 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.527157 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.527297 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.528835 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:38.528841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:38.528844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:38.528865 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:38.528900 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:38.528913 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:38.528998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.529258 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.529352 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260807 11:21:38.530786 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.518805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:41015 (local address 127.2.223.131:57017)
0807 11:21:38.519781 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.519828 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.521631 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.522466 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.522516 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.522633 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:38.523467 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.523496 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.524805 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.524810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.526327 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.526341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.527408 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.527415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.527679 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.528456 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:38.528458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:38.528504 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:38.528604 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:38.529011 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:38.529019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:38.529026 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.529187 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.529523 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
I20260807 11:21:38.532447 23373 heartbeater.cc:347] Connected to a master server at 127.2.223.190:41015
I20260807 11:21:38.532792 23373 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:38.533609 23373 heartbeater.cc:511] Master 127.2.223.190:41015 requested a full tablet report, sending...
I20260807 11:21:38.535277 22902 ts_manager.cc:194] Registered new tserver with Master: 49eb39376bed4898845c028406ff2fdc (127.2.223.131:42223)
I20260807 11:21:38.536137 22902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.131:57017
I20260807 11:21:38.540202 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:38.551349 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.544589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.2.223.190:41015)
0807 11:21:38.544810 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.544818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.545278 (+ 460us) server_negotiation.cc:407] Connection header received
0807 11:21:38.545520 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.545524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.545593 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.545724 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:38.546245 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.547060 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.548517 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.548941 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.549098 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.550426 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:38.550431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:38.550434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:38.550456 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:38.550540 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:38.550559 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:38.550724 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.550948 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.551012 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:21:38.555929 22902 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:38.563310 22902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:38.598239 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.578377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.2.223.130:46179)
0807 11:21:38.586698 (+ 8321us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.586715 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.588914 (+ 2199us) server_negotiation.cc:407] Connection header received
0807 11:21:38.589087 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.589100 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.589526 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.589853 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:38.591092 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.592647 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.594610 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.596363 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.596527 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.596835 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.597642 (+ 807us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.597858 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8032,"thread_start_us":7878,"threads_started":1}
I20260807 11:21:38.598206 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.577793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:46179 (local address 127.0.0.1:46052)
0807 11:21:38.587457 (+ 9664us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.587496 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.587779 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.590265 (+ 2486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.590277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.590325 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:38.590864 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.590880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.592793 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.592801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.594327 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.594342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.597004 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.597012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.597311 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.597400 (+ 89us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.597560 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.597810 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9433,"thread_start_us":10129,"threads_started":1}
I20260807 11:21:38.600080 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.578913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:42223 (local address 127.0.0.1:56998)
0807 11:21:38.588054 (+ 9141us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.588096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.588375 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.591908 (+ 3533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.591914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.591954 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:38.592368 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.592381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.594611 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.594618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.597115 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.597131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.599131 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.599139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.599339 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.599376 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.599480 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.599622 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8910,"mutex_wait_us":1,"thread_start_us":9591,"threads_started":1}
I20260807 11:21:38.600411 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.577784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.2.223.129:42575)
0807 11:21:38.590414 (+ 12630us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.590426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.590467 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:21:38.590652 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.590664 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.590964 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.592177 (+ 1213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:38.592480 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.594320 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.597022 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.598637 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.599459 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.599774 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.599860 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.600059 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12409,"thread_start_us":12253,"threads_started":1}
I20260807 11:21:38.600808 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.579837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.2.223.131:42223)
0807 11:21:38.590915 (+ 11078us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.590925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.590966 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:21:38.591075 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.591087 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.591320 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.591575 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:38.592626 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.594341 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.597365 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.598878 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.599023 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.599335 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.600110 (+ 775us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.600314 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10878,"thread_start_us":10685,"threads_started":1}
I20260807 11:21:38.601333 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.577290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43860)
0807 11:21:38.578210 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.578258 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.578452 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.591262 (+ 12810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.591269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.591303 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:38.591882 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.591897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.594536 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.594551 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.596806 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.596821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.598889 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.598896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.599102 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.599133 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.599199 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.599355 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":123,"thread_start_us":383,"threads_started":1}
I20260807 11:21:38.603567 23164 tablet_service.cc:1467] Processing CreateTablet for tablet 4c64cf7ed678468cb51958bf03977d94 (DEFAULT_TABLE table=test-table [id=5f93cb80df4f463b81472f1ccf3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:38.604651 23304 tablet_service.cc:1467] Processing CreateTablet for tablet 4c64cf7ed678468cb51958bf03977d94 (DEFAULT_TABLE table=test-table [id=5f93cb80df4f463b81472f1ccf3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:38.605327 23164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c64cf7ed678468cb51958bf03977d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:38.605408 23028 tablet_service.cc:1467] Processing CreateTablet for tablet 4c64cf7ed678468cb51958bf03977d94 (DEFAULT_TABLE table=test-table [id=5f93cb80df4f463b81472f1ccf3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:38.606137 23304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c64cf7ed678468cb51958bf03977d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:38.607434 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c64cf7ed678468cb51958bf03977d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:38.621718 23391 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Bootstrap starting.
I20260807 11:21:38.625289 23391 tablet_bootstrap.cc:654] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:38.626641 23394 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Bootstrap starting.
I20260807 11:21:38.627179 23391 log.cc:824] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:38.629390 23395 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Bootstrap starting.
I20260807 11:21:38.631198 23394 tablet_bootstrap.cc:654] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:38.632761 23394 log.cc:824] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:38.633104 23391 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: No bootstrap required, opened a new log
I20260807 11:21:38.633486 23391 ts_tablet_manager.cc:1397] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 11:21:38.633488 23395 tablet_bootstrap.cc:654] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:38.634663 23395 log.cc:824] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:38.636431 23394 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: No bootstrap required, opened a new log
I20260807 11:21:38.636745 23394 ts_tablet_manager.cc:1397] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:21:38.638167 23395 tablet_bootstrap.cc:492] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: No bootstrap required, opened a new log
I20260807 11:21:38.638473 23395 ts_tablet_manager.cc:1397] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 11:21:38.643345 23391 raft_consensus.cc:348] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.643803 23391 raft_consensus.cc:374] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:38.643952 23391 raft_consensus.cc:729] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49eb39376bed4898845c028406ff2fdc, State: Initialized, Role: FOLLOWER
I20260807 11:21:38.644728 23391 consensus_queue.cc:260] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.646142 23373 heartbeater.cc:503] Master 127.2.223.190:41015 was elected leader, sending a full tablet report...
I20260807 11:21:38.646991 23391 ts_tablet_manager.cc:1428] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260807 11:21:38.647962 23394 raft_consensus.cc:348] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.648449 23394 raft_consensus.cc:374] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:38.648574 23394 raft_consensus.cc:729] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a483363da1584710b3f495d177e2f423, State: Initialized, Role: FOLLOWER
I20260807 11:21:38.649240 23394 consensus_queue.cc:260] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.650430 23235 heartbeater.cc:503] Master 127.2.223.190:41015 was elected leader, sending a full tablet report...
I20260807 11:21:38.650985 23395 raft_consensus.cc:348] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.651257 23394 ts_tablet_manager.cc:1428] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260807 11:21:38.651340 23395 raft_consensus.cc:374] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:38.651407 23395 raft_consensus.cc:729] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83b13e8eeaf4e4789222ec48d486eed, State: Initialized, Role: FOLLOWER
I20260807 11:21:38.652076 23395 consensus_queue.cc:260] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.654080 23395 ts_tablet_manager.cc:1428] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
W20260807 11:21:38.727322 23098 tablet.cc:2369] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:38.758818 23236 tablet.cc:2369] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:38.767371 23374 tablet.cc:2369] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:38.952744 23399 raft_consensus.cc:482] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:38.953121 23399 raft_consensus.cc:504] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.955284 23399 leader_election.cc:288] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a83b13e8eeaf4e4789222ec48d486eed (127.2.223.129:42575), a483363da1584710b3f495d177e2f423 (127.2.223.130:46179)
I20260807 11:21:38.963007 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.955734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:55607 (local address 127.2.223.129:42575)
0807 11:21:38.955968 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.955974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.956296 (+ 322us) server_negotiation.cc:407] Connection header received
0807 11:21:38.956415 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.956421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.956519 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.956655 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:38.957128 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.958212 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.960916 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.961967 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.962155 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.962211 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.962592 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.962751 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:21:38.963055 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.955696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.2.223.131:55607)
0807 11:21:38.956064 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.956088 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.956258 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.956720 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.956725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.956751 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:38.956916 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.956924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.958388 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.958392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.960720 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.960745 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.962153 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.962159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.962363 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.962405 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.962561 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.962690 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1}
I20260807 11:21:38.963268 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.955845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:46179 (local address 127.2.223.131:36011)
0807 11:21:38.956470 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:38.956485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:38.956600 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:38.957016 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:38.957030 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:38.957058 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:38.957349 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.957356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.958738 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.958743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.960813 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:38.960826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.962391 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.962397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:38.962611 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.962644 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:21:38.962736 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 11:21:38.962871 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":28,"thread_start_us":342,"threads_started":1}
I20260807 11:21:38.963217 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:38.955969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:36011 (local address 127.2.223.130:46179)
0807 11:21:38.956224 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:38.956230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:38.956592 (+ 362us) server_negotiation.cc:407] Connection header received
0807 11:21:38.956779 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:38.956788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:38.956881 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:38.956964 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:38.957518 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.958596 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.961036 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:38.962209 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:38.962374 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:38.962437 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:38.962805 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 11:21:38.962948 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:21:38.964485 23050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c64cf7ed678468cb51958bf03977d94" candidate_uuid: "49eb39376bed4898845c028406ff2fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b13e8eeaf4e4789222ec48d486eed" is_pre_election: true
I20260807 11:21:38.964486 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c64cf7ed678468cb51958bf03977d94" candidate_uuid: "49eb39376bed4898845c028406ff2fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a483363da1584710b3f495d177e2f423" is_pre_election: true
I20260807 11:21:38.965081 23186 raft_consensus.cc:2454] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49eb39376bed4898845c028406ff2fdc in term 0.
I20260807 11:21:38.965065 23050 raft_consensus.cc:2454] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49eb39376bed4898845c028406ff2fdc in term 0.
I20260807 11:21:38.965813 23258 leader_election.cc:302] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49eb39376bed4898845c028406ff2fdc, a483363da1584710b3f495d177e2f423; no voters:
I20260807 11:21:38.966274 23399 raft_consensus.cc:2781] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:38.966457 23399 raft_consensus.cc:482] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:38.966610 23399 raft_consensus.cc:3037] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:38.968959 23399 raft_consensus.cc:504] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.969645 23399 leader_election.cc:288] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [CANDIDATE]: Term 1 election: Requested vote from peers a83b13e8eeaf4e4789222ec48d486eed (127.2.223.129:42575), a483363da1584710b3f495d177e2f423 (127.2.223.130:46179)
I20260807 11:21:38.970243 23050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c64cf7ed678468cb51958bf03977d94" candidate_uuid: "49eb39376bed4898845c028406ff2fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b13e8eeaf4e4789222ec48d486eed"
I20260807 11:21:38.970311 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c64cf7ed678468cb51958bf03977d94" candidate_uuid: "49eb39376bed4898845c028406ff2fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a483363da1584710b3f495d177e2f423"
I20260807 11:21:38.970534 23050 raft_consensus.cc:3037] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:38.970633 23186 raft_consensus.cc:3037] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:38.972985 23050 raft_consensus.cc:2454] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49eb39376bed4898845c028406ff2fdc in term 1.
I20260807 11:21:38.973040 23186 raft_consensus.cc:2454] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49eb39376bed4898845c028406ff2fdc in term 1.
I20260807 11:21:38.973580 23257 leader_election.cc:302] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49eb39376bed4898845c028406ff2fdc, a83b13e8eeaf4e4789222ec48d486eed; no voters:
I20260807 11:21:38.973964 23399 raft_consensus.cc:2781] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:38.974427 23399 raft_consensus.cc:686] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 1 LEADER]: Becoming Leader. State: Replica: 49eb39376bed4898845c028406ff2fdc, State: Running, Role: LEADER
I20260807 11:21:38.975085 23399 consensus_queue.cc:237] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } }
I20260807 11:21:38.980593 22902 catalog_manager.cc:5495] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 49eb39376bed4898845c028406ff2fdc (127.2.223.131). New cstate: current_term: 1 leader_uuid: "49eb39376bed4898845c028406ff2fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49eb39376bed4898845c028406ff2fdc" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 42223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 } health_report { overall_health: UNKNOWN } } }
I20260807 11:21:39.064666 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.058453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.2.223.131:42223)
0807 11:21:39.058703 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:39.058709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:39.059132 (+ 423us) server_negotiation.cc:407] Connection header received
0807 11:21:39.059378 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:39.059383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:39.059473 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:39.059677 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:39.060122 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.060917 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.062431 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.062962 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.063071 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:39.063594 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:39.064115 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:39.064253 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:39.064402 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 11:21:39.064441 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 11:21:39.073818 23050 raft_consensus.cc:1261] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed [term 1 FOLLOWER]: Refusing update from remote peer 49eb39376bed4898845c028406ff2fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:39.073822 23186 raft_consensus.cc:1261] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 1 FOLLOWER]: Refusing update from remote peer 49eb39376bed4898845c028406ff2fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:39.074986 23403 consensus_queue.cc:1035] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83b13e8eeaf4e4789222ec48d486eed" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:39.075525 23399 consensus_queue.cc:1035] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a483363da1584710b3f495d177e2f423" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:39.094791 23408 mvcc.cc:205] Tried to move back new op lower bound from 7315872559395815424 to 7315872559006277632. Current Snapshot: MvccSnapshot[applied={T|T < 7315872559395815424}]
I20260807 11:21:39.106297 22901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:39.106622 22901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:39.110534 22901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c86dec0e048405fa08c4aa5661dfc1d: Sending DeleteTablet for 3 replicas of tablet 4c64cf7ed678468cb51958bf03977d94
I20260807 11:21:39.111989 23304 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c64cf7ed678468cb51958bf03977d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:39 UTC) from {username='slave'} at 127.0.0.1:56998
I20260807 11:21:39.111981 23028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c64cf7ed678468cb51958bf03977d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:39 UTC) from {username='slave'} at 127.0.0.1:43860
I20260807 11:21:39.112116 23164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c64cf7ed678468cb51958bf03977d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:39 UTC) from {username='slave'} at 127.0.0.1:46052
I20260807 11:21:39.114557 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22964
I20260807 11:21:39.124406 23419 tablet_replica.cc:330] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: stopping tablet replica
I20260807 11:21:39.125098 23419 raft_consensus.cc:2229] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:39.125735 23419 raft_consensus.cc:2258] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:39.125826 23418 tablet_replica.cc:330] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: stopping tablet replica
I20260807 11:21:39.126362 23418 raft_consensus.cc:2229] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:39.127475 23418 raft_consensus.cc:2258] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:39.130234 23419 ts_tablet_manager.cc:1905] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:39.130252 23418 ts_tablet_manager.cc:1905] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:39.132874 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.116626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.2.223.129:42575)
0807 11:21:39.116828 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:39.116835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:39.121907 (+ 5072us) server_negotiation.cc:407] Connection header received
0807 11:21:39.122649 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:39.122659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:39.122821 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:39.122925 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:39.124548 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.125737 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.128316 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.129060 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.129188 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:39.130967 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:39.130980 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:39.131000 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:39.131861 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:39.131930 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:39.131959 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:39.132054 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:39.132346 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 11:21:39.132484 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:21:39.134234 23091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:39.141403 23418 ts_tablet_manager.cc:1918] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:39.141734 23418 log.cc:1196] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/4c64cf7ed678468cb51958bf03977d94
I20260807 11:21:39.142457 23418 ts_tablet_manager.cc:1939] T 4c64cf7ed678468cb51958bf03977d94 P 49eb39376bed4898845c028406ff2fdc: Deleting consensus metadata
I20260807 11:21:39.144716 22888 catalog_manager.cc:4841] TS 49eb39376bed4898845c028406ff2fdc (127.2.223.131:42223): tablet 4c64cf7ed678468cb51958bf03977d94 (table test-table [id=5f93cb80df4f463b81472f1ccf3dc31c]) successfully deleted
I20260807 11:21:39.148516 23419 ts_tablet_manager.cc:1918] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:39.148748 23419 log.cc:1196] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/4c64cf7ed678468cb51958bf03977d94
I20260807 11:21:39.149314 23419 ts_tablet_manager.cc:1939] T 4c64cf7ed678468cb51958bf03977d94 P a483363da1584710b3f495d177e2f423: Deleting consensus metadata
I20260807 11:21:39.151216 22889 catalog_manager.cc:4841] TS a483363da1584710b3f495d177e2f423 (127.2.223.130:46179): tablet 4c64cf7ed678468cb51958bf03977d94 (table test-table [id=5f93cb80df4f463b81472f1ccf3dc31c]) successfully deleted
I20260807 11:21:39.290789 23417 tablet_replica.cc:330] T 4c64cf7ed678468cb51958bf03977d94 P a83b13e8eeaf4e4789222ec48d486eed: stopping tablet replica
I20260807 11:21:39.292173 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22964
I20260807 11:21:39.325321 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23101
W20260807 11:21:39.325809 22888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.223.129:42575 (error 108)
I20260807 11:21:39.336910 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.327498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.2.223.130:46179)
0807 11:21:39.327701 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:39.327707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:39.327755 (+ 48us) server_negotiation.cc:407] Connection header received
0807 11:21:39.327829 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:39.327836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:39.327950 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:39.328078 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:39.328792 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.329995 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.332057 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.332736 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.332868 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:39.334469 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:39.334504 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:39.334527 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:39.335421 (+ 894us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:39.335492 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:39.335520 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:39.335645 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:39.336361 (+ 716us) server_negotiation.cc:299] Negotiation successful
0807 11:21:39.336499 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:21:39.337945 23230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:39.344468 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.328290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43880)
0807 11:21:39.342212 (+ 13922us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.344146 (+ 1934us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13719,"thread_start_us":15344,"threads_started":1}
W20260807 11:21:39.344977 22888 catalog_manager.cc:4584] TS a83b13e8eeaf4e4789222ec48d486eed (127.2.223.129:42575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c64cf7ed678468cb51958bf03977d94: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
I20260807 11:21:39.399410 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.398801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43890)
0807 11:21:39.399117 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.399225 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260807 11:21:39.400797 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.400322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43902)
0807 11:21:39.400531 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.400612 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 11:21:39.441212 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23101
I20260807 11:21:39.461714 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.461022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43906)
0807 11:21:39.461378 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.461495 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260807 11:21:39.463317 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.462762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43912)
0807 11:21:39.463004 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.463103 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260807 11:21:39.464856 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23239
I20260807 11:21:39.472463 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.465616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.2.223.131:42223)
0807 11:21:39.465793 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:39.465799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:39.465934 (+ 135us) server_negotiation.cc:407] Connection header received
0807 11:21:39.466157 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:39.466163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:39.466263 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:39.466390 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:39.466881 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.467682 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.469031 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:39.469516 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:39.469661 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:39.470927 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:39.470938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:39.470955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:39.471581 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:39.471641 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:39.471662 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:39.471808 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:39.471942 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 11:21:39.472053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:21:39.473214 23368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:39.543926 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.543285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43918)
0807 11:21:39.543614 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.543743 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260807 11:21:39.545207 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.544708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43920)
0807 11:21:39.544933 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.545019 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260807 11:21:39.592365 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23239
I20260807 11:21:39.616534 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 22869
I20260807 11:21:39.617766 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:39.732471 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:39.731876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:42575 (local address 127.0.0.1:43930)
0807 11:21:39.732132 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:39.732249 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.129:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 11:21:39.732740 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 22869
2026-08-07T11:21:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3519 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:39 dist-test-slave-5l3k krb5kdc[23429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:39 dist-test-slave-5l3k krb5kdc[23429](info): set up 2 sockets
Aug 07 11:21:39 dist-test-slave-5l3k krb5kdc[23429](info): commencing operation
krb5kdc: starting...
W20260807 11:21:40.936513 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:41 dist-test-slave-5l3k krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:21:41.325423 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46187
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:41.696316 23444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:41.696754 23444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:41.696928 23444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:41.704655 23444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:41.704849 23444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:41.704965 23444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:41.705072 23444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:41.705152 23444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:41.715863 23444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:41.717427 23444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:41.718991 23444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:41.735196 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:41.735196 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:41.735351 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:41.737526 23444 server_base.cc:1034] running on GCE node
I20260807 11:21:41.738839 23444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:41.741120 23444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:41.742447 23444 hybrid_clock.cc:648] HybridClock initialized: now 1786101701742354 us; error 87 us; skew 500 ppm
I20260807 11:21:41.743000 23444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:41 dist-test-slave-5l3k krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:41.747599 23444 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:21:41.749892 23444 webserver.cc:460] Webserver started at http://127.2.223.190:42819/ using document root <none> and password file <none>
I20260807 11:21:41.751041 23444 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:41.751287 23444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:41.751699 23444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:41.756145 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "6c255299f719421b909deaa5c0c6c4b6"
format_stamp: "Formatted at 2026-08-07 11:21:41 on dist-test-slave-5l3k"
server_key: "f9573d52f748187892e2a27470f9ae1f"
server_key_iv: "1d9e26a465f26289cf4c195cfe1bce5d"
server_key_version: "encryptionkey@0"
I20260807 11:21:41.757323 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "6c255299f719421b909deaa5c0c6c4b6"
format_stamp: "Formatted at 2026-08-07 11:21:41 on dist-test-slave-5l3k"
server_key: "f9573d52f748187892e2a27470f9ae1f"
server_key_iv: "1d9e26a465f26289cf4c195cfe1bce5d"
server_key_version: "encryptionkey@0"
I20260807 11:21:41.766059 23444 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260807 11:21:41.770947 23459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:41.772733 23444 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 11:21:41.772969 23444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "6c255299f719421b909deaa5c0c6c4b6"
format_stamp: "Formatted at 2026-08-07 11:21:41 on dist-test-slave-5l3k"
server_key: "f9573d52f748187892e2a27470f9ae1f"
server_key_iv: "1d9e26a465f26289cf4c195cfe1bce5d"
server_key_version: "encryptionkey@0"
I20260807 11:21:41.773213 23444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:41.791749 23444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:41.796546 23444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:41.796896 23444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:41.817566 23444 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:46187
I20260807 11:21:41.817586 23514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:46187 every 8 connection(s)
I20260807 11:21:41.819146 23444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:41.823376 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:41.823907 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23444
I20260807 11:21:41.824100 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:41.824561 2942 external_mini_cluster.cc:1428] Setting key d37d1778dd623252b8c8885e5ad38435
I20260807 11:21:41.839115 23515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: Bootstrap starting.
Aug 07 11:21:41 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:41.842936 23515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:41.844693 23515 log.cc:824] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:41.848886 23515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: No bootstrap required, opened a new log
I20260807 11:21:41.853212 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:41.826604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.2.223.190:46187)
0807 11:21:41.827551 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:41.827567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:41.827628 (+ 61us) server_negotiation.cc:407] Connection header received
0807 11:21:41.830668 (+ 3040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:41.830704 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:41.831332 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:41.831832 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:41.833415 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:41.835545 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:41.836934 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:41.838076 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:41.838276 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:41.842900 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:41.842920 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:41.842944 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:41.842998 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:41.847629 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:41.848613 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:41.848622 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:41.848631 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:41.848885 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:41.849327 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:41.849332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:41.849335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:41.850173 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:41.850546 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:41.850986 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 11:21:41.851570 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":398,"threads_started":1}
I20260807 11:21:41.858834 23515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } }
I20260807 11:21:41.859318 23515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:41.859439 23515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c255299f719421b909deaa5c0c6c4b6, State: Initialized, Role: FOLLOWER
I20260807 11:21:41.860051 23515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } }
I20260807 11:21:41.860321 23515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:41.860484 23515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:41.860682 23515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:41.863315 23515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } }
I20260807 11:21:41.863893 23515 leader_election.cc:302] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c255299f719421b909deaa5c0c6c4b6; no voters:
I20260807 11:21:41.864604 23515 leader_election.cc:288] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:41.864753 23521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:41.865496 23521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6c255299f719421b909deaa5c0c6c4b6, State: Running, Role: LEADER
I20260807 11:21:41.866158 23521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } }
I20260807 11:21:41.867136 23515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:41.869745 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c255299f719421b909deaa5c0c6c4b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } } }
I20260807 11:21:41.869717 23523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c255299f719421b909deaa5c0c6c4b6. Latest consensus state: current_term: 1 leader_uuid: "6c255299f719421b909deaa5c0c6c4b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c255299f719421b909deaa5c0c6c4b6" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46187 } } }
I20260807 11:21:41.870394 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:41.870419 23523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:41.872712 23528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:41.878039 23528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:41.888305 23528 catalog_manager.cc:1347] Generated new cluster ID: 57f69dc2ef224e2f8259eeefb908ef3f
I20260807 11:21:41.888463 23528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:41.898240 23528 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:41.899724 23528 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:41.908010 23528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: Generated new TSK 0
I20260807 11:21:41.909142 23528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.223.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
Entry for principal kudu/127.2.223.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab.
I20260807 11:21:42.003551 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:42.380923 23543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:42.381261 23543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:42.381362 23543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:42.390298 23543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:42.390444 23543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:42.390674 23543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:42.401295 23543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.129.keytab
--principal=kudu/127.2.223.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:42.402704 23543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:42.404062 23543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:42.418524 23549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:42.418342 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:42.418525 23552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:42.419682 23543 server_base.cc:1034] running on GCE node
I20260807 11:21:42.420473 23543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:42.421629 23543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:42.422871 23543 hybrid_clock.cc:648] HybridClock initialized: now 1786101702422813 us; error 50 us; skew 500 ppm
I20260807 11:21:42.423243 23543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:42 dist-test-slave-5l3k krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:42.426819 23543 init.cc:377] Logged in from keytab as kudu/127.2.223.129@KRBTEST.COM (short username kudu)
I20260807 11:21:42.428565 23543 webserver.cc:460] Webserver started at http://127.2.223.129:39143/ using document root <none> and password file <none>
I20260807 11:21:42.429338 23543 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:42.429497 23543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:42.429845 23543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:42.433383 23543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "b57dd314cfb745599ac306bb1f80fca4"
format_stamp: "Formatted at 2026-08-07 11:21:42 on dist-test-slave-5l3k"
server_key: "a583ff5c55b8ca606c3f443b165f6f84"
server_key_iv: "6184acfbe0003acdb088a4b95bcc3831"
server_key_version: "encryptionkey@0"
I20260807 11:21:42.434296 23543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "b57dd314cfb745599ac306bb1f80fca4"
format_stamp: "Formatted at 2026-08-07 11:21:42 on dist-test-slave-5l3k"
server_key: "a583ff5c55b8ca606c3f443b165f6f84"
server_key_iv: "6184acfbe0003acdb088a4b95bcc3831"
server_key_version: "encryptionkey@0"
I20260807 11:21:42.440550 23543 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:42.444586 23560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:42.446064 23543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:21:42.446292 23543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "b57dd314cfb745599ac306bb1f80fca4"
format_stamp: "Formatted at 2026-08-07 11:21:42 on dist-test-slave-5l3k"
server_key: "a583ff5c55b8ca606c3f443b165f6f84"
server_key_iv: "6184acfbe0003acdb088a4b95bcc3831"
server_key_version: "encryptionkey@0"
I20260807 11:21:42.446581 23543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:42.464885 23543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:42.469702 23543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:42.470064 23543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:42.471253 23543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:42.472904 23543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:42.473023 23543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:42.473222 23543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:42.473300 23543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:42.510572 23543 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:32793
I20260807 11:21:42.510634 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:32793 every 8 connection(s)
I20260807 11:21:42.512035 23543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:42.513556 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23543
I20260807 11:21:42.513762 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:42.514364 2942 external_mini_cluster.cc:1428] Setting key 8fa9d5767f92e04a46156e113c7545ae
Aug 07 11:21:42 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.129@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:42.539595 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:42.515230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:35323 (local address 127.2.223.190:46187)
0807 11:21:42.515751 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:42.515756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:42.516986 (+ 1230us) server_negotiation.cc:407] Connection header received
0807 11:21:42.519091 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:42.519101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:42.519205 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:42.519347 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:42.520951 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:42.522098 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:42.524782 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:42.525522 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:42.525688 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:42.531342 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:42.531353 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:42.531362 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:42.531421 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:42.535567 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:42.536473 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:42.536490 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:42.536496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:42.536683 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:42.537198 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:42.537206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:42.537212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:42.537681 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:42.537881 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:42.538741 (+ 860us) server_negotiation.cc:299] Negotiation successful
0807 11:21:42.539052 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260807 11:21:42.540279 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:42.515846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46187 (local address 127.2.223.129:35323)
0807 11:21:42.516804 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:42.516855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:42.518795 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:42.519582 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:42.519591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:42.520151 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:42.520757 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:42.520776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:42.522364 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:42.522370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:42.524587 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:42.524613 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:42.525838 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:42.525847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:42.526294 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:42.527299 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:42.527343 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:42.531017 (+ 3674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:42.535762 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:42.535771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:42.535787 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:42.536296 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:42.536826 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:42.536829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:42.536832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:42.537057 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:42.537839 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:42.537846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:42.538239 (+ 393us) client_negotiation.cc:769] Sending connection context
0807 11:21:42.538574 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 11:21:42.539041 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":421,"threads_started":1}
I20260807 11:21:42.542057 23675 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46187
I20260807 11:21:42.542456 23675 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:42.543593 23675 heartbeater.cc:511] Master 127.2.223.190:46187 requested a full tablet report, sending...
I20260807 11:21:42.546151 23479 ts_manager.cc:194] Registered new tserver with Master: b57dd314cfb745599ac306bb1f80fca4 (127.2.223.129:32793)
I20260807 11:21:42.547729 23479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.129@KRBTEST.COM'} at 127.2.223.129:35323
WARNING: no policy specified for kudu/127.2.223.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
Entry for principal kudu/127.2.223.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab.
I20260807 11:21:42.600577 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:42.977916 23682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:42.978235 23682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:42.978411 23682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:42.986205 23682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:42.986340 23682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:42.986557 23682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:42.996927 23682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.130.keytab
--principal=kudu/127.2.223.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:42.998412 23682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:43.000312 23682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:43.015021 23690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:43.014932 23692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:43.014994 23689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:43.016319 23682 server_base.cc:1034] running on GCE node
I20260807 11:21:43.017093 23682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:43.018187 23682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:43.019456 23682 hybrid_clock.cc:648] HybridClock initialized: now 1786101703019380 us; error 69 us; skew 500 ppm
I20260807 11:21:43.019830 23682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:43 dist-test-slave-5l3k krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:43.023334 23682 init.cc:377] Logged in from keytab as kudu/127.2.223.130@KRBTEST.COM (short username kudu)
I20260807 11:21:43.025019 23682 webserver.cc:460] Webserver started at http://127.2.223.130:36083/ using document root <none> and password file <none>
I20260807 11:21:43.025794 23682 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:43.025952 23682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:43.026302 23682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:43.029579 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "8820cde44e6240949ea86d13e22c2ff0"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "da11fdeb749ed5b2bea57d75db8a7792"
server_key_iv: "091a40b769b0357b86f6a91b6aa65b9d"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.030439 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "8820cde44e6240949ea86d13e22c2ff0"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "da11fdeb749ed5b2bea57d75db8a7792"
server_key_iv: "091a40b769b0357b86f6a91b6aa65b9d"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.036260 23682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:21:43.040370 23699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.041779 23682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:21:43.041972 23682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "8820cde44e6240949ea86d13e22c2ff0"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "da11fdeb749ed5b2bea57d75db8a7792"
server_key_iv: "091a40b769b0357b86f6a91b6aa65b9d"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.042196 23682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:43.072726 23682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:43.077364 23682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:43.077702 23682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:43.078867 23682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:43.080528 23682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:43.080646 23682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.080809 23682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:43.080895 23682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.117295 23682 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:36635
I20260807 11:21:43.117347 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:36635 every 8 connection(s)
I20260807 11:21:43.118790 23682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:43.119915 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23682
I20260807 11:21:43.120105 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:43.120675 2942 external_mini_cluster.cc:1428] Setting key f03bd7c15eb4ff98948f575ff1a05db8
Aug 07 11:21:43 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.130@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:43.145660 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.121903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:55111 (local address 127.2.223.190:46187)
0807 11:21:43.122444 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.122451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.123602 (+ 1151us) server_negotiation.cc:407] Connection header received
0807 11:21:43.126767 (+ 3165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.126771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.126850 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.126952 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:43.129379 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.130215 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.132904 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.133361 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.133469 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.138340 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:43.138346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:43.138349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:43.138390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:43.142035 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.143012 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.143019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.143022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.143180 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.143697 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.143703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.143707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.144047 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:43.144217 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.144991 (+ 774us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.145248 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260807 11:21:43.147149 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.122565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46187 (local address 127.2.223.130:55111)
0807 11:21:43.123409 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:43.123464 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:43.126357 (+ 2893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:43.127283 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:43.127298 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:43.128197 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:43.129121 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.129165 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.130409 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.130418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.132719 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.132739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.133625 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.133633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.134015 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.134917 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:43.134956 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:43.138030 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:43.142245 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:43.142254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:43.142271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:43.142852 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:43.143309 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:43.143314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:43.143324 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:43.143557 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:43.144190 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:43.144199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:43.144597 (+ 398us) client_negotiation.cc:769] Sending connection context
0807 11:21:43.144888 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 11:21:43.145517 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":341,"threads_started":1}
I20260807 11:21:43.148839 23818 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46187
I20260807 11:21:43.149230 23818 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:43.150274 23818 heartbeater.cc:511] Master 127.2.223.190:46187 requested a full tablet report, sending...
I20260807 11:21:43.152200 23479 ts_manager.cc:194] Registered new tserver with Master: 8820cde44e6240949ea86d13e22c2ff0 (127.2.223.130:36635)
I20260807 11:21:43.153147 23479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.130@KRBTEST.COM'} at 127.2.223.130:55111
WARNING: no policy specified for kudu/127.2.223.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
Entry for principal kudu/127.2.223.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab.
I20260807 11:21:43.210302 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:21:43.552132 23675 heartbeater.cc:503] Master 127.2.223.190:46187 was elected leader, sending a full tablet report...
W20260807 11:21:43.595754 23824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:43.596109 23824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:43.596295 23824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:43.604182 23824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:43.604343 23824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:43.604555 23824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:43.615386 23824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.131.keytab
--principal=kudu/127.2.223.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:43.616882 23824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:43.618418 23824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:43.633524 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:43.633621 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:43.633621 23833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:43.634550 23824 server_base.cc:1034] running on GCE node
I20260807 11:21:43.635353 23824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:43.636487 23824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:43.637763 23824 hybrid_clock.cc:648] HybridClock initialized: now 1786101703637695 us; error 53 us; skew 500 ppm
I20260807 11:21:43.638162 23824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:43 dist-test-slave-5l3k krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:43.646057 23824 init.cc:377] Logged in from keytab as kudu/127.2.223.131@KRBTEST.COM (short username kudu)
I20260807 11:21:43.648306 23824 webserver.cc:460] Webserver started at http://127.2.223.131:46405/ using document root <none> and password file <none>
I20260807 11:21:43.649396 23824 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:43.649647 23824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:43.650040 23824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:43.653699 23824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "e47e2ec361424b2a996ac7d0ddd1af44"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "83643df374905d81cc79e19462af90e4"
server_key_iv: "ff556153a25e183083a58a0028f5c0f1"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.654785 23824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "e47e2ec361424b2a996ac7d0ddd1af44"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "83643df374905d81cc79e19462af90e4"
server_key_iv: "ff556153a25e183083a58a0028f5c0f1"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.661429 23824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 11:21:43.665926 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.667909 23824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:21:43.668157 23824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "e47e2ec361424b2a996ac7d0ddd1af44"
format_stamp: "Formatted at 2026-08-07 11:21:43 on dist-test-slave-5l3k"
server_key: "83643df374905d81cc79e19462af90e4"
server_key_iv: "ff556153a25e183083a58a0028f5c0f1"
server_key_version: "encryptionkey@0"
I20260807 11:21:43.668478 23824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:43.699926 23824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:43.705353 23824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:43.705804 23824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:43.707343 23824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:43.708839 23824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:43.708922 23824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.709012 23824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:43.709077 23824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:43.747841 23824 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:33597
I20260807 11:21:43.747948 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:33597 every 8 connection(s)
I20260807 11:21:43.749356 23824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:43.751883 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 23824
I20260807 11:21:43.752105 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:43.752666 2942 external_mini_cluster.cc:1428] Setting key a94e17d95eba77abe653cbbe4885bace
Aug 07 11:21:43 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.131@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:21:43.773218 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.752740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:40073 (local address 127.2.223.190:46187)
0807 11:21:43.753323 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.753328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.757062 (+ 3734us) server_negotiation.cc:407] Connection header received
0807 11:21:43.757121 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.757125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.757203 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.757294 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:43.759311 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.760116 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.761813 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.762236 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.762339 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.766323 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:43.766329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:43.766333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:43.766373 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:43.769796 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.770760 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.770764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.770766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.770904 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.771371 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.771374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.771376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.771664 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:43.771799 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.772615 (+ 816us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.772909 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
I20260807 11:21:43.774235 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.753547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46187 (local address 127.2.223.131:40073)
0807 11:21:43.754607 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:43.754662 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:43.756717 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:43.757628 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:43.757642 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:43.758331 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:43.759108 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.759132 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.760301 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.760305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.761653 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.761668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.762466 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.762472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.762785 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.763518 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:21:43.763544 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:43.766013 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:43.769969 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:43.769979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:43.769995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:43.770633 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:43.771014 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:21:43.771017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:21:43.771020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:21:43.771261 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:21:43.771788 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:43.771794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:43.772218 (+ 424us) client_negotiation.cc:769] Sending connection context
0807 11:21:43.772466 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 11:21:43.772919 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":450,"threads_started":1}
I20260807 11:21:43.775961 23956 heartbeater.cc:347] Connected to a master server at 127.2.223.190:46187
I20260807 11:21:43.776286 23956 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:43.777177 23956 heartbeater.cc:511] Master 127.2.223.190:46187 requested a full tablet report, sending...
I20260807 11:21:43.778931 23479 ts_manager.cc:194] Registered new tserver with Master: e47e2ec361424b2a996ac7d0ddd1af44 (127.2.223.131:33597)
I20260807 11:21:43.780051 23479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.223.131@KRBTEST.COM'} at 127.2.223.131:40073
I20260807 11:21:43.782021 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:43.802299 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.787634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.2.223.190:46187)
0807 11:21:43.787825 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.787853 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.788333 (+ 480us) server_negotiation.cc:407] Connection header received
0807 11:21:43.788557 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.788563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.788653 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.788771 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:43.790285 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.791180 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.792562 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.792997 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.793167 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.796123 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:43.796129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:43.796133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:43.796176 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:43.799153 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.799986 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.799994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.800002 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.800149 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:43.800633 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:43.800639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:43.800642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:43.801013 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:43.801291 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.801679 (+ 388us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.801947 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:21:43.806906 23479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:43.814656 23479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:43.852028 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.830187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:32793 (local address 127.0.0.1:38148)
0807 11:21:43.839417 (+ 9230us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:43.839491 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:43.839754 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:43.843077 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:43.843085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:43.843123 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:43.843756 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.843775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.846104 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.846109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.848673 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.848697 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.850690 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.850697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.850994 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.851061 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 11:21:43.851365 (+ 304us) client_negotiation.cc:240] Negotiation successful
0807 11:21:43.851603 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8848,"thread_start_us":9109,"threads_started":1}
I20260807 11:21:43.852666 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.830904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.2.223.130:36635)
0807 11:21:43.843297 (+ 12393us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.843311 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.843336 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:21:43.843454 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.843464 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.843752 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.843999 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:43.845030 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.846734 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.849551 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.851072 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.851265 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.851475 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.852150 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.852338 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":12113,"threads_started":1}
I20260807 11:21:43.853049 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.843217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33597 (local address 127.0.0.1:37802)
0807 11:21:43.843981 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:43.844002 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:43.844237 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:43.845039 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:43.845044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:43.845085 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:43.845454 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.845465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.847604 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.847608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.849836 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.849855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.852218 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.852226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.852391 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.852424 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:21:43.852519 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 11:21:43.852686 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":393,"threads_started":1}
I20260807 11:21:43.853138 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.830449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:36635 (local address 127.0.0.1:52970)
0807 11:21:43.841282 (+ 10833us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:43.841305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:43.841448 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:43.844316 (+ 2868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:43.844322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:43.844349 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:43.844792 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.844806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.846895 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.846905 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.849346 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:43.849361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.851325 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.851332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:43.851526 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.851561 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:21:43.851666 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:21:43.851825 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10685,"mutex_wait_us":9,"thread_start_us":9642,"threads_started":1}
I20260807 11:21:43.853879 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.841270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.2.223.131:33597)
0807 11:21:43.842777 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.842790 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.844114 (+ 1324us) server_negotiation.cc:407] Connection header received
0807 11:21:43.844458 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.844468 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.844726 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.844920 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:43.845650 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.847400 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.850103 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.851703 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.851824 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.852043 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.853385 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.853598 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":1008,"threads_started":1}
I20260807 11:21:43.854344 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:43.831070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.2.223.129:32793)
0807 11:21:43.842219 (+ 11149us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:43.842234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:43.842278 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:21:43.842407 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:43.842416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:43.842767 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:43.844003 (+ 1236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:43.844308 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.845931 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.848920 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:43.850430 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:43.850623 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:43.850897 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:43.851601 (+ 704us) server_negotiation.cc:299] Negotiation successful
0807 11:21:43.851805 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10879,"thread_start_us":10675,"threads_started":1}
I20260807 11:21:43.857252 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 (DEFAULT_TABLE table=test-table [id=90294b4f314d4691b2be44822a05b330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:43.857677 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 (DEFAULT_TABLE table=test-table [id=90294b4f314d4691b2be44822a05b330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:43.858240 23608 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 (DEFAULT_TABLE table=test-table [id=90294b4f314d4691b2be44822a05b330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:43.858927 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78df8cd05f4b3a83c920ace9da1e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:43.859349 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78df8cd05f4b3a83c920ace9da1e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:43.859705 23608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78df8cd05f4b3a83c920ace9da1e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:43.873108 23975 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Bootstrap starting.
I20260807 11:21:43.876549 23976 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Bootstrap starting.
I20260807 11:21:43.877171 23975 tablet_bootstrap.cc:654] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:43.878643 23977 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Bootstrap starting.
I20260807 11:21:43.878885 23975 log.cc:824] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:43.881083 23976 tablet_bootstrap.cc:654] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:43.882469 23977 tablet_bootstrap.cc:654] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:43.882896 23976 log.cc:824] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:43.883845 23977 log.cc:824] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:43.887578 23975 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: No bootstrap required, opened a new log
I20260807 11:21:43.887756 23977 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: No bootstrap required, opened a new log
I20260807 11:21:43.887840 23976 tablet_bootstrap.cc:492] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: No bootstrap required, opened a new log
I20260807 11:21:43.887984 23975 ts_tablet_manager.cc:1397] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260807 11:21:43.888077 23977 ts_tablet_manager.cc:1397] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:21:43.888077 23976 ts_tablet_manager.cc:1397] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260807 11:21:43.896389 23975 raft_consensus.cc:348] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.896893 23975 raft_consensus.cc:374] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:43.897013 23975 raft_consensus.cc:729] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8820cde44e6240949ea86d13e22c2ff0, State: Initialized, Role: FOLLOWER
I20260807 11:21:43.897758 23975 consensus_queue.cc:260] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.899088 23818 heartbeater.cc:503] Master 127.2.223.190:46187 was elected leader, sending a full tablet report...
I20260807 11:21:43.899607 23977 raft_consensus.cc:348] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.899612 23976 raft_consensus.cc:348] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.899972 23975 ts_tablet_manager.cc:1428] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260807 11:21:43.900105 23976 raft_consensus.cc:374] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:43.900110 23977 raft_consensus.cc:374] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:43.900214 23977 raft_consensus.cc:729] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57dd314cfb745599ac306bb1f80fca4, State: Initialized, Role: FOLLOWER
I20260807 11:21:43.900218 23976 raft_consensus.cc:729] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47e2ec361424b2a996ac7d0ddd1af44, State: Initialized, Role: FOLLOWER
I20260807 11:21:43.901026 23976 consensus_queue.cc:260] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.901032 23977 consensus_queue.cc:260] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:43.903340 23977 ts_tablet_manager.cc:1428] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260807 11:21:43.903463 23976 ts_tablet_manager.cc:1428] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260807 11:21:43.904311 23956 heartbeater.cc:503] Master 127.2.223.190:46187 was elected leader, sending a full tablet report...
W20260807 11:21:44.001550 23957 tablet.cc:2369] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:44.015105 23676 tablet.cc:2369] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:44.052565 23982 raft_consensus.cc:482] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:44.052927 23982 raft_consensus.cc:504] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:44.054952 23982 leader_election.cc:288] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57dd314cfb745599ac306bb1f80fca4 (127.2.223.129:32793), e47e2ec361424b2a996ac7d0ddd1af44 (127.2.223.131:33597)
I20260807 11:21:44.062269 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.055428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:51693 (local address 127.2.223.129:32793)
0807 11:21:44.055650 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.055654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.056373 (+ 719us) server_negotiation.cc:407] Connection header received
0807 11:21:44.056655 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.056661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.056755 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.056877 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:44.057591 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.058372 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.060341 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.061175 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.061389 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.061486 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.061815 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.061960 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:21:44.062376 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.055438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:32793 (local address 127.2.223.130:51693)
0807 11:21:44.056221 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:44.056245 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:44.056404 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:44.056980 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:44.056985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:44.057018 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:44.057423 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:44.057434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.058554 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.058563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:44.060138 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:44.060153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.061466 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.061471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:44.061639 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.061666 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 11:21:44.061782 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 11:21:44.061947 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":421,"threads_started":1}
I20260807 11:21:44.062779 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.056641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:52373 (local address 127.2.223.131:33597)
0807 11:21:44.056875 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.056880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.057232 (+ 352us) server_negotiation.cc:407] Connection header received
0807 11:21:44.057451 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.057455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.057517 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.057620 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:44.058208 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.059076 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.060910 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.061727 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.061928 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.062008 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.062342 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.062512 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:21:44.062917 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.056394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:33597 (local address 127.2.223.130:52373)
0807 11:21:44.057103 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:44.057123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:44.057294 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:44.057634 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:44.057639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:44.057662 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:44.058021 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:44.058031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.059231 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.059238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:44.060716 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:44.060730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.061967 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.061973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:44.062162 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.062186 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:21:44.062309 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 11:21:44.062457 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":2,"thread_start_us":374,"threads_started":1}
I20260807 11:21:44.064121 23911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78df8cd05f4b3a83c920ace9da1e06" candidate_uuid: "8820cde44e6240949ea86d13e22c2ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" is_pre_election: true
I20260807 11:21:44.064117 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78df8cd05f4b3a83c920ace9da1e06" candidate_uuid: "8820cde44e6240949ea86d13e22c2ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57dd314cfb745599ac306bb1f80fca4" is_pre_election: true
I20260807 11:21:44.064777 23629 raft_consensus.cc:2454] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8820cde44e6240949ea86d13e22c2ff0 in term 0.
I20260807 11:21:44.064777 23911 raft_consensus.cc:2454] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8820cde44e6240949ea86d13e22c2ff0 in term 0.
I20260807 11:21:44.065623 23701 leader_election.cc:302] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8820cde44e6240949ea86d13e22c2ff0, e47e2ec361424b2a996ac7d0ddd1af44; no voters:
I20260807 11:21:44.066104 23982 raft_consensus.cc:2781] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:44.066273 23982 raft_consensus.cc:482] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:44.066408 23982 raft_consensus.cc:3037] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:44.069051 23982 raft_consensus.cc:504] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:44.069737 23982 leader_election.cc:288] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [CANDIDATE]: Term 1 election: Requested vote from peers b57dd314cfb745599ac306bb1f80fca4 (127.2.223.129:32793), e47e2ec361424b2a996ac7d0ddd1af44 (127.2.223.131:33597)
I20260807 11:21:44.070420 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78df8cd05f4b3a83c920ace9da1e06" candidate_uuid: "8820cde44e6240949ea86d13e22c2ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57dd314cfb745599ac306bb1f80fca4"
I20260807 11:21:44.070644 23911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78df8cd05f4b3a83c920ace9da1e06" candidate_uuid: "8820cde44e6240949ea86d13e22c2ff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47e2ec361424b2a996ac7d0ddd1af44"
I20260807 11:21:44.070732 23629 raft_consensus.cc:3037] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:44.070904 23911 raft_consensus.cc:3037] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:44.073462 23629 raft_consensus.cc:2454] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8820cde44e6240949ea86d13e22c2ff0 in term 1.
I20260807 11:21:44.073716 23911 raft_consensus.cc:2454] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8820cde44e6240949ea86d13e22c2ff0 in term 1.
I20260807 11:21:44.074136 23701 leader_election.cc:302] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8820cde44e6240949ea86d13e22c2ff0, b57dd314cfb745599ac306bb1f80fca4; no voters:
I20260807 11:21:44.074568 23982 raft_consensus.cc:2781] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:44.075104 23982 raft_consensus.cc:686] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 1 LEADER]: Becoming Leader. State: Replica: 8820cde44e6240949ea86d13e22c2ff0, State: Running, Role: LEADER
I20260807 11:21:44.075688 23982 consensus_queue.cc:237] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } }
I20260807 11:21:44.081099 23478 catalog_manager.cc:5495] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8820cde44e6240949ea86d13e22c2ff0 (127.2.223.130). New cstate: current_term: 1 leader_uuid: "8820cde44e6240949ea86d13e22c2ff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8820cde44e6240949ea86d13e22c2ff0" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 } health_report { overall_health: UNKNOWN } } }
W20260807 11:21:44.121357 23819 tablet.cc:2369] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:44.153240 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.147429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.2.223.130:36635)
0807 11:21:44.147609 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.147614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.147633 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:21:44.147813 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.147817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.147899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.148033 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:44.148395 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.149144 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.150791 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.151296 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.151425 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.151997 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:44.152555 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:44.152678 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.152794 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.152872 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 11:21:44.163082 23911 raft_consensus.cc:1261] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 1 FOLLOWER]: Refusing update from remote peer 8820cde44e6240949ea86d13e22c2ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:44.163082 23629 raft_consensus.cc:1261] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 1 FOLLOWER]: Refusing update from remote peer 8820cde44e6240949ea86d13e22c2ff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:44.164332 23987 consensus_queue.cc:1035] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47e2ec361424b2a996ac7d0ddd1af44" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:44.164878 23982 consensus_queue.cc:1035] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57dd314cfb745599ac306bb1f80fca4" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 32793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:44.203601 23479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:44.203989 23479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:44.208405 23479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c255299f719421b909deaa5c0c6c4b6: Sending DeleteTablet for 3 replicas of tablet 5c78df8cd05f4b3a83c920ace9da1e06
I20260807 11:21:44.209868 23608 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:38148
I20260807 11:21:44.209971 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:52970
I20260807 11:21:44.211532 24002 tablet_replica.cc:330] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: stopping tablet replica
I20260807 11:21:44.212146 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78df8cd05f4b3a83c920ace9da1e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:44 UTC) from {username='kudu', principal='kudu/127.2.223.190@KRBTEST.COM'} at 127.0.0.1:37802
I20260807 11:21:44.212222 24002 raft_consensus.cc:2229] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:44.212947 24002 raft_consensus.cc:2258] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:44.216288 24002 ts_tablet_manager.cc:1905] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:44.217006 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23543
I20260807 11:21:44.227375 24002 ts_tablet_manager.cc:1918] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:44.227661 24002 log.cc:1196] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/5c78df8cd05f4b3a83c920ace9da1e06
I20260807 11:21:44.228395 24002 ts_tablet_manager.cc:1939] T 5c78df8cd05f4b3a83c920ace9da1e06 P b57dd314cfb745599ac306bb1f80fca4: Deleting consensus metadata
I20260807 11:21:44.231325 23461 catalog_manager.cc:4841] TS b57dd314cfb745599ac306bb1f80fca4 (127.2.223.129:32793): tablet 5c78df8cd05f4b3a83c920ace9da1e06 (table test-table [id=90294b4f314d4691b2be44822a05b330]) successfully deleted
I20260807 11:21:44.233464 24004 tablet_replica.cc:330] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: stopping tablet replica
I20260807 11:21:44.234174 24004 raft_consensus.cc:2229] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:44.234997 24004 raft_consensus.cc:2258] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:44.236049 24003 tablet_replica.cc:330] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: stopping tablet replica
I20260807 11:21:44.236627 24003 raft_consensus.cc:2229] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:44.237540 24003 raft_consensus.cc:2258] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:44.238644 24004 ts_tablet_manager.cc:1905] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:44.240046 24003 ts_tablet_manager.cc:1905] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:21:44 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.129@KRBTEST.COM
I20260807 11:21:44.246877 24004 ts_tablet_manager.cc:1918] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:44.247279 24004 log.cc:1196] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/5c78df8cd05f4b3a83c920ace9da1e06
I20260807 11:21:44.247961 24004 ts_tablet_manager.cc:1939] T 5c78df8cd05f4b3a83c920ace9da1e06 P e47e2ec361424b2a996ac7d0ddd1af44: Deleting consensus metadata
I20260807 11:21:44.248118 24003 ts_tablet_manager.cc:1918] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:44.248324 24003 log.cc:1196] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/5c78df8cd05f4b3a83c920ace9da1e06
I20260807 11:21:44.248921 24003 ts_tablet_manager.cc:1939] T 5c78df8cd05f4b3a83c920ace9da1e06 P 8820cde44e6240949ea86d13e22c2ff0: Deleting consensus metadata
I20260807 11:21:44.250602 23461 catalog_manager.cc:4841] TS e47e2ec361424b2a996ac7d0ddd1af44 (127.2.223.131:33597): tablet 5c78df8cd05f4b3a83c920ace9da1e06 (table test-table [id=90294b4f314d4691b2be44822a05b330]) successfully deleted
I20260807 11:21:44.251025 23460 catalog_manager.cc:4841] TS 8820cde44e6240949ea86d13e22c2ff0 (127.2.223.130:36635): tablet 5c78df8cd05f4b3a83c920ace9da1e06 (table test-table [id=90294b4f314d4691b2be44822a05b330]) successfully deleted
I20260807 11:21:44.254606 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.218823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.2.223.129:32793)
0807 11:21:44.219043 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.219049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.232510 (+ 13461us) server_negotiation.cc:407] Connection header received
0807 11:21:44.232775 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.232782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.232890 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.233004 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:44.235624 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.236795 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.239261 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.239956 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.240095 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.246338 (+ 6243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:44.246363 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:44.246385 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:44.246444 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:44.251074 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.252009 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.252031 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.252036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.252248 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.252693 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.252698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.252702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.253241 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:44.253582 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.253806 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.254098 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:21:44.255478 23670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:44.379907 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23543
I20260807 11:21:44.410240 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23682
Aug 07 11:21:44 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.130@KRBTEST.COM
I20260807 11:21:44.430261 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.411192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.2.223.130:36635)
0807 11:21:44.411399 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.411405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.411430 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:21:44.411502 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.411508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.411601 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.411728 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:44.413232 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.414537 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.415883 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.416635 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.416783 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.422206 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:44.422232 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:44.422265 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:44.422347 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:44.426954 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.427833 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.427845 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.427849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.428040 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.428439 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.428445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.428448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.428981 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:44.429332 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.429569 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.429831 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 11:21:44.431176 23813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:44.553778 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23682
I20260807 11:21:44.580771 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23824
Aug 07 11:21:44 dist-test-slave-5l3k krb5kdc[23429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.131@KRBTEST.COM
I20260807 11:21:44.598703 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:44.581602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.2.223.131:33597)
0807 11:21:44.582155 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:44.582160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:44.582177 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:21:44.582235 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:44.582239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:44.582303 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:44.582444 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:44.583985 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.584740 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.586160 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:44.586700 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:44.586852 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:44.591990 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:44.592010 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:44.592030 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:44.592079 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:44.595318 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.596219 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.596228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.596230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.596406 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:44.596855 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:44.596863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:44.596870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:44.597282 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:44.597617 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:44.597950 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 11:21:44.598239 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260807 11:21:44.599506 23951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:44.694459 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23824
I20260807 11:21:44.717392 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 23444
I20260807 11:21:44.718706 23510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:44.837947 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 23444
2026-08-07T11:21:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T11:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:44Z Disabled control of system clock
I20260807 11:21:44.950773 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36435
--webserver_interface=127.2.223.190
--webserver_port=0
--builtin_ntp_servers=127.2.223.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:36435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:45.331044 24016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:45.331416 24016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:45.331606 24016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:45.339589 24016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:45.339731 24016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:45.339833 24016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:45.339955 24016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:45.340024 24016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:45.350070 24016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:36435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:45.351644 24016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:45.353663 24016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:45.366968 24022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:45.366921 24025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:45.367012 24023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:45.368517 24016 server_base.cc:1034] running on GCE node
I20260807 11:21:45.369535 24016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:45.371268 24016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:45.372496 24016 hybrid_clock.cc:648] HybridClock initialized: now 1786101705372449 us; error 38 us; skew 500 ppm
I20260807 11:21:45.372846 24016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:45.375454 24016 webserver.cc:460] Webserver started at http://127.2.223.190:36617/ using document root <none> and password file <none>
I20260807 11:21:45.376204 24016 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:45.376304 24016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:45.376608 24016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:45.380052 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "409e9e1bef7447e383124a80ea3c25bf"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "fac162069e24e27846efe748c21da7a0"
server_key_iv: "bbdf767ff2c1742f122809c5d6357027"
server_key_version: "encryptionkey@0"
I20260807 11:21:45.380924 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "409e9e1bef7447e383124a80ea3c25bf"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "fac162069e24e27846efe748c21da7a0"
server_key_iv: "bbdf767ff2c1742f122809c5d6357027"
server_key_version: "encryptionkey@0"
I20260807 11:21:45.387135 24016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 11:21:45.391382 24031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:45.393054 24016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:21:45.393311 24016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "409e9e1bef7447e383124a80ea3c25bf"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "fac162069e24e27846efe748c21da7a0"
server_key_iv: "bbdf767ff2c1742f122809c5d6357027"
server_key_version: "encryptionkey@0"
I20260807 11:21:45.393585 24016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:45.423619 24016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:45.424649 24016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:45.425009 24016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:45.446336 24016 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:36435
I20260807 11:21:45.446354 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:36435 every 8 connection(s)
I20260807 11:21:45.447999 24016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:45.451027 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24016
I20260807 11:21:45.451243 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:45.451754 2942 external_mini_cluster.cc:1428] Setting key d0eb482cb40ec8526cc5cd62e8378d8a
I20260807 11:21:45.453008 24086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:45.469870 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:45.454035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.2.223.190:36435)
0807 11:21:45.455387 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:45.455405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:45.455472 (+ 67us) server_negotiation.cc:407] Connection header received
0807 11:21:45.457387 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:45.457427 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:45.458320 (+ 893us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:45.458928 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:45.459610 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:45.461842 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:45.463521 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:45.464660 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:45.464875 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:45.466176 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:45.466189 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:45.466219 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:45.467147 (+ 928us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:45.467274 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:45.467307 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:45.467462 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:45.467732 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 11:21:45.468192 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":480,"threads_started":1}
I20260807 11:21:45.471205 24086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: Bootstrap starting.
I20260807 11:21:45.475289 24086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:45.476400 24086 log.cc:824] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:45.479786 24086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: No bootstrap required, opened a new log
I20260807 11:21:45.487748 24086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } }
I20260807 11:21:45.488250 24086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:45.488379 24086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409e9e1bef7447e383124a80ea3c25bf, State: Initialized, Role: FOLLOWER
I20260807 11:21:45.489101 24086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } }
I20260807 11:21:45.489375 24086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:45.489528 24086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:45.489741 24086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:45.492411 24086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } }
I20260807 11:21:45.493103 24086 leader_election.cc:302] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 409e9e1bef7447e383124a80ea3c25bf; no voters:
I20260807 11:21:45.493793 24086 leader_election.cc:288] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:45.494050 24091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:45.495007 24091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [term 1 LEADER]: Becoming Leader. State: Replica: 409e9e1bef7447e383124a80ea3c25bf, State: Running, Role: LEADER
I20260807 11:21:45.495674 24091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } }
I20260807 11:21:45.496367 24086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:45.499246 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 409e9e1bef7447e383124a80ea3c25bf. Latest consensus state: current_term: 1 leader_uuid: "409e9e1bef7447e383124a80ea3c25bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } } }
I20260807 11:21:45.499745 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:45.500968 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "409e9e1bef7447e383124a80ea3c25bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409e9e1bef7447e383124a80ea3c25bf" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 36435 } } }
I20260807 11:21:45.501368 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:45.502362 24098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:45.507675 24098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:45.518293 24098 catalog_manager.cc:1347] Generated new cluster ID: 08b1c3d6011b4c8fa909c10fa727bdef
I20260807 11:21:45.518460 24098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:45.535449 24098 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:45.537470 24098 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:45.552055 24098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: Generated new TSK 0
I20260807 11:21:45.553372 24098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:45.563197 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.129:0
--local_ip_for_outbound_sockets=127.2.223.129
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--builtin_ntp_servers=127.2.223.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:21:45.953928 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:45.954294 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:45.954391 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:45.962553 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:45.962679 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:45.962798 24110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.129
I20260807 11:21:45.973134 24110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.223.129
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.129
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:45.974665 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:45.976667 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:45.991583 24117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:45.991816 24116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:45.991782 24120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:45.992905 24110 server_base.cc:1034] running on GCE node
I20260807 11:21:45.993625 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:45.994808 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:45.996083 24110 hybrid_clock.cc:648] HybridClock initialized: now 1786101705996026 us; error 50 us; skew 500 ppm
I20260807 11:21:45.996500 24110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:45.999526 24110 webserver.cc:460] Webserver started at http://127.2.223.129:42967/ using document root <none> and password file <none>
I20260807 11:21:46.000391 24110 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:46.000548 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:46.000939 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:46.004535 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data/instance:
uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "0498c19913f44e0f83bd0c314f9444fe"
server_key_iv: "40d6c79cab5baf47a8c65e6330191a2b"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.005538 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance:
uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "0498c19913f44e0f83bd0c314f9444fe"
server_key_iv: "40d6c79cab5baf47a8c65e6330191a2b"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.011582 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:21:46.015515 24128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.017146 24110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:21:46.017381 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal
uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a"
format_stamp: "Formatted at 2026-08-07 11:21:45 on dist-test-slave-5l3k"
server_key: "0498c19913f44e0f83bd0c314f9444fe"
server_key_iv: "40d6c79cab5baf47a8c65e6330191a2b"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.017658 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:46.035189 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:46.036181 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:46.036540 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:46.037662 24110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:46.039495 24110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:46.039597 24110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.039767 24110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:46.039871 24110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.077929 24110 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.129:41119
I20260807 11:21:46.077984 24248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.129:41119 every 8 connection(s)
I20260807 11:21:46.079541 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/data/info.pb
I20260807 11:21:46.084292 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24110
I20260807 11:21:46.084484 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal/instance
I20260807 11:21:46.085129 2942 external_mini_cluster.cc:1428] Setting key 2eb2ebb339de6425a997261b65be6ed4
I20260807 11:21:46.088320 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.130:0
--local_ip_for_outbound_sockets=127.2.223.130
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--builtin_ntp_servers=127.2.223.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:46.094615 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:46.082615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:35449 (local address 127.2.223.190:36435)
0807 11:21:46.083141 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:46.083148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:46.084881 (+ 1733us) server_negotiation.cc:407] Connection header received
0807 11:21:46.087207 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:46.087213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:46.087313 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:46.087448 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:46.088786 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.089555 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.091404 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.091826 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.091984 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:46.093602 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:46.093607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:46.093611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:46.093633 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:46.093678 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:46.093691 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:46.093801 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:46.094064 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 11:21:46.094182 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":242,"threads_started":1}
I20260807 11:21:46.095794 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:46.083340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36435 (local address 127.2.223.129:35449)
0807 11:21:46.084654 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:46.084723 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:46.086880 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:46.087746 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:46.087757 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:46.087867 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:46.088576 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:46.088604 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.089735 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.089747 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:46.091227 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:46.091243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.092098 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.092106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:46.092389 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:46.093207 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:46.093210 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:46.093259 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:46.093357 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:46.093802 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:46.093817 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:46.093828 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:21:46.094043 (+ 215us) client_negotiation.cc:240] Negotiation successful
0807 11:21:46.094562 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":377,"threads_started":1}
I20260807 11:21:46.097880 24249 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36435
I20260807 11:21:46.098305 24249 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:46.099292 24249 heartbeater.cc:511] Master 127.2.223.190:36435 requested a full tablet report, sending...
I20260807 11:21:46.101965 24049 ts_manager.cc:194] Registered new tserver with Master: fe6c9b8f3c7547cd8f79e70c5112f69a (127.2.223.129:41119)
I20260807 11:21:46.104285 24049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.129:35449
W20260807 11:21:46.483651 24253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:46.484081 24253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:46.484246 24253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:46.492312 24253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:46.492501 24253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:46.492692 24253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.130
I20260807 11:21:46.503216 24253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.223.130
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.130
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:46.504714 24253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:46.506910 24253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:46.521831 24258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:46.522244 24259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:46.522766 24261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:46.523563 24253 server_base.cc:1034] running on GCE node
I20260807 11:21:46.524269 24253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:46.525452 24253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:46.526706 24253 hybrid_clock.cc:648] HybridClock initialized: now 1786101706526675 us; error 76 us; skew 500 ppm
I20260807 11:21:46.527063 24253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:46.530354 24253 webserver.cc:460] Webserver started at http://127.2.223.130:34845/ using document root <none> and password file <none>
I20260807 11:21:46.531509 24253 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:46.531657 24253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:46.532127 24253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:46.537844 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data/instance:
uuid: "9a34aa898db3492f8af58b49a84d4f15"
format_stamp: "Formatted at 2026-08-07 11:21:46 on dist-test-slave-5l3k"
server_key: "ab494e155f5f3aab5655b347f7404a11"
server_key_iv: "ec16b3b33672dd5d6f0029b1349593d9"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.539484 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance:
uuid: "9a34aa898db3492f8af58b49a84d4f15"
format_stamp: "Formatted at 2026-08-07 11:21:46 on dist-test-slave-5l3k"
server_key: "ab494e155f5f3aab5655b347f7404a11"
server_key_iv: "ec16b3b33672dd5d6f0029b1349593d9"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.546759 24253 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260807 11:21:46.550901 24267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.552551 24253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:21:46.552794 24253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal
uuid: "9a34aa898db3492f8af58b49a84d4f15"
format_stamp: "Formatted at 2026-08-07 11:21:46 on dist-test-slave-5l3k"
server_key: "ab494e155f5f3aab5655b347f7404a11"
server_key_iv: "ec16b3b33672dd5d6f0029b1349593d9"
server_key_version: "encryptionkey@0"
I20260807 11:21:46.553074 24253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:46.577775 24253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:46.578910 24253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:46.579293 24253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:46.580623 24253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:46.583179 24253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:46.583334 24253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.583499 24253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:46.583598 24253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:46.623382 24253 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.130:36643
I20260807 11:21:46.623500 24385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.130:36643 every 8 connection(s)
I20260807 11:21:46.625264 24253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/data/info.pb
I20260807 11:21:46.632160 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24253
I20260807 11:21:46.632390 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal/instance
I20260807 11:21:46.632982 2942 external_mini_cluster.cc:1428] Setting key 8163643f757510817c7f996ddd6a603b
I20260807 11:21:46.636679 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.223.131:0
--local_ip_for_outbound_sockets=127.2.223.131
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--builtin_ntp_servers=127.2.223.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:21:46.643010 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:46.628371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.130:43699 (local address 127.2.223.190:36435)
0807 11:21:46.628904 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:46.628909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:46.630453 (+ 1544us) server_negotiation.cc:407] Connection header received
0807 11:21:46.632969 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:46.632976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:46.633096 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:46.633235 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:46.634836 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.635912 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.638752 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.639523 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.639715 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:46.641799 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:46.641808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:46.641816 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:46.641849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:46.641909 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:46.641930 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:46.642033 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:46.642435 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 11:21:46.642605 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260807 11:21:46.644520 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:46.629168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36435 (local address 127.2.223.130:43699)
0807 11:21:46.630258 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:46.630303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:46.632538 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:46.633558 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:46.633571 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:46.633728 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:46.634636 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:46.634661 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.636135 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.636141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:46.638461 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:46.638526 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:46.639862 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:46.639870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:46.640274 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:46.641262 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:46.641266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:46.641333 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:46.641471 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:46.642095 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:46.642112 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:46.642126 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 11:21:46.642398 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 11:21:46.642970 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260807 11:21:46.646593 24387 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36435
I20260807 11:21:46.646950 24387 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:46.647794 24387 heartbeater.cc:511] Master 127.2.223.190:36435 requested a full tablet report, sending...
I20260807 11:21:46.649550 24049 ts_manager.cc:194] Registered new tserver with Master: 9a34aa898db3492f8af58b49a84d4f15 (127.2.223.130:36643)
I20260807 11:21:46.650820 24049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.130:43699
W20260807 11:21:47.092388 24391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:47.092734 24391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:47.092914 24391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:47.101977 24391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:47.102113 24391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:47.102309 24391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.223.131
I20260807 11:21:47.108927 24249 heartbeater.cc:503] Master 127.2.223.190:36435 was elected leader, sending a full tablet report...
I20260807 11:21:47.113711 24391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.223.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.223.131
--webserver_port=0
--tserver_master_addrs=127.2.223.190:36435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.223.131
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:47.115391 24391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:47.117519 24391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:47.133438 24397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:47.133431 24396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:47.133855 24399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:47.134999 24391 server_base.cc:1034] running on GCE node
I20260807 11:21:47.135924 24391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:47.137133 24391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:47.138417 24391 hybrid_clock.cc:648] HybridClock initialized: now 1786101707138349 us; error 64 us; skew 500 ppm
I20260807 11:21:47.138896 24391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:21:47.141765 24391 webserver.cc:460] Webserver started at http://127.2.223.131:44673/ using document root <none> and password file <none>
I20260807 11:21:47.142716 24391 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:47.142879 24391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:47.143260 24391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:47.147856 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data/instance:
uuid: "95a5a92cc335458c9a0b4adde1d66a6b"
format_stamp: "Formatted at 2026-08-07 11:21:47 on dist-test-slave-5l3k"
server_key: "3599b49ca49f740075e2497abb584141"
server_key_iv: "e43c41fa5235ffdfc2ce952f15d6c191"
server_key_version: "encryptionkey@0"
I20260807 11:21:47.149101 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance:
uuid: "95a5a92cc335458c9a0b4adde1d66a6b"
format_stamp: "Formatted at 2026-08-07 11:21:47 on dist-test-slave-5l3k"
server_key: "3599b49ca49f740075e2497abb584141"
server_key_iv: "e43c41fa5235ffdfc2ce952f15d6c191"
server_key_version: "encryptionkey@0"
I20260807 11:21:47.157299 24391 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260807 11:21:47.162002 24405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:47.163848 24391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:21:47.164095 24391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal
uuid: "95a5a92cc335458c9a0b4adde1d66a6b"
format_stamp: "Formatted at 2026-08-07 11:21:47 on dist-test-slave-5l3k"
server_key: "3599b49ca49f740075e2497abb584141"
server_key_iv: "e43c41fa5235ffdfc2ce952f15d6c191"
server_key_version: "encryptionkey@0"
I20260807 11:21:47.164382 24391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:47.184059 24391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:47.185081 24391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:47.185459 24391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:47.186690 24391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:21:47.188614 24391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:21:47.188755 24391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:47.188884 24391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:21:47.188975 24391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:47.229734 24391 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.131:45577
I20260807 11:21:47.229805 24521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.131:45577 every 8 connection(s)
I20260807 11:21:47.231359 24391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/data/info.pb
I20260807 11:21:47.237946 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24391
I20260807 11:21:47.238152 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal/instance
I20260807 11:21:47.238705 2942 external_mini_cluster.cc:1428] Setting key 1fb39eb68eb55e2a5fc8635091726b6b
I20260807 11:21:47.248449 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.235356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.131:42905 (local address 127.2.223.190:36435)
0807 11:21:47.235960 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.235967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.237524 (+ 1557us) server_negotiation.cc:407] Connection header received
0807 11:21:47.240316 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.240322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.240421 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.240554 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:47.242367 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.243328 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.245114 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.245573 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.245744 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.247455 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:47.247460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:47.247464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:47.247487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:47.247535 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:47.247548 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:47.247668 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.247920 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.248043 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":313,"threads_started":1}
I20260807 11:21:47.249684 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.236046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:36435 (local address 127.2.223.131:42905)
0807 11:21:47.237259 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.237328 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.239983 (+ 2655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.241061 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.241075 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.241213 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:21:47.242145 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.242200 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.243533 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.243537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.244930 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.244946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.245870 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.245881 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.246173 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.247039 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:21:47.247042 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:21:47.247091 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:21:47.247206 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:21:47.247659 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:21:47.247672 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:21:47.247684 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.247910 (+ 226us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.248425 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":340,"threads_started":1}
I20260807 11:21:47.251663 24522 heartbeater.cc:347] Connected to a master server at 127.2.223.190:36435
I20260807 11:21:47.252089 24522 heartbeater.cc:464] Registering TS with master...
I20260807 11:21:47.252987 24522 heartbeater.cc:511] Master 127.2.223.190:36435 requested a full tablet report, sending...
I20260807 11:21:47.254983 24049 ts_manager.cc:194] Registered new tserver with Master: 95a5a92cc335458c9a0b4adde1d66a6b (127.2.223.131:45577)
I20260807 11:21:47.255753 2942 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:21:47.256198 24049 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.223.131:42905
I20260807 11:21:47.268889 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.261109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.2.223.190:36435)
0807 11:21:47.261367 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.261376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.261888 (+ 512us) server_negotiation.cc:407] Connection header received
0807 11:21:47.262203 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.262207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.262276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.262367 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:47.263266 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.263984 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.266150 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.266728 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.266856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.268118 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:47.268123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:47.268126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:47.268151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:47.268200 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:47.268214 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:47.268292 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.268455 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.268552 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:21:47.274037 24049 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:21:47.281638 24049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:21:47.311712 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.298075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.2.223.131:45577)
0807 11:21:47.300564 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.300575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.300618 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:21:47.300739 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.300941 (+ 202us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.301402 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.301680 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:47.303370 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.305070 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.308136 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.309591 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.309759 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.310143 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.310932 (+ 789us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.311158 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2284,"thread_start_us":2116,"threads_started":1}
I20260807 11:21:47.312676 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.297377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:45577 (local address 127.0.0.1:51494)
0807 11:21:47.298185 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.298222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.298379 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.302519 (+ 4140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.302529 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.302579 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:47.303079 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.303098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.305306 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.305311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.307897 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.307911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.310281 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.310288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.310606 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.310671 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.312066 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.312310 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":3,"thread_start_us":462,"threads_started":1}
I20260807 11:21:47.313338 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.300224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:36643 (local address 127.0.0.1:50696)
0807 11:21:47.302601 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.302628 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.302801 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.303879 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.303883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.303911 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:47.304313 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.304327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.306437 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.306446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.309391 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.309415 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.312315 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.312325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.312609 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.312671 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.312753 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.312892 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2156,"thread_start_us":1944,"threads_started":1}
I20260807 11:21:47.315014 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.300775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.2.223.130:36643)
0807 11:21:47.303349 (+ 2574us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.303358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.303390 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:21:47.303478 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.303484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.303674 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.303889 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:47.304523 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.306234 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.309813 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.311915 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.313781 (+ 1866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.314162 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.314256 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.314521 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2426,"thread_start_us":2323,"threads_started":1}
I20260807 11:21:47.317617 24453 tablet_service.cc:1467] Processing CreateTablet for tablet 1786304873014a7da3bc0535b2c55d53 (DEFAULT_TABLE table=test-table [id=c215c416959c4cf2aa6383813437663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:47.319416 24453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1786304873014a7da3bc0535b2c55d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:47.319550 24318 tablet_service.cc:1467] Processing CreateTablet for tablet 1786304873014a7da3bc0535b2c55d53 (DEFAULT_TABLE table=test-table [id=c215c416959c4cf2aa6383813437663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:47.321162 24318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1786304873014a7da3bc0535b2c55d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:47.322865 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.304659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.129:41119 (local address 127.0.0.1:52158)
0807 11:21:47.313063 (+ 8404us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.313086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.313218 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.314445 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.314451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.314500 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:47.314682 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.314696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.316776 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.316782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.318765 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.318789 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.321285 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.321293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.321519 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.321635 (+ 116us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.321734 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.321910 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8291,"thread_start_us":14256,"threads_started":1}
I20260807 11:21:47.323410 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.305317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.2.223.129:41119)
0807 11:21:47.307548 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.307561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.313381 (+ 5820us) server_negotiation.cc:407] Connection header received
0807 11:21:47.313612 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.313622 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.313900 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.314131 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:47.314866 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.316568 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.319025 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.320650 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.320808 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.321115 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.322247 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.322455 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1922,"thread_start_us":1770,"threads_started":1}
I20260807 11:21:47.334766 24541 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Bootstrap starting.
I20260807 11:21:47.334909 24179 tablet_service.cc:1467] Processing CreateTablet for tablet 1786304873014a7da3bc0535b2c55d53 (DEFAULT_TABLE table=test-table [id=c215c416959c4cf2aa6383813437663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:21:47.336699 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1786304873014a7da3bc0535b2c55d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:47.339004 24541 tablet_bootstrap.cc:654] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:47.339272 24542 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Bootstrap starting.
I20260807 11:21:47.340435 24541 log.cc:824] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:47.345209 24542 tablet_bootstrap.cc:654] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:47.346706 24542 log.cc:824] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:47.348060 24541 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: No bootstrap required, opened a new log
I20260807 11:21:47.348573 24541 ts_tablet_manager.cc:1397] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260807 11:21:47.358568 24542 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: No bootstrap required, opened a new log
I20260807 11:21:47.360731 24542 ts_tablet_manager.cc:1397] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20260807 11:21:47.361418 24545 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Bootstrap starting.
I20260807 11:21:47.361361 24541 raft_consensus.cc:348] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.362226 24541 raft_consensus.cc:374] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:47.362418 24541 raft_consensus.cc:729] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a5a92cc335458c9a0b4adde1d66a6b, State: Initialized, Role: FOLLOWER
I20260807 11:21:47.363356 24541 consensus_queue.cc:260] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.364908 24522 heartbeater.cc:503] Master 127.2.223.190:36435 was elected leader, sending a full tablet report...
I20260807 11:21:47.366207 24541 ts_tablet_manager.cc:1428] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260807 11:21:47.368185 24545 tablet_bootstrap.cc:654] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:47.369825 24545 log.cc:824] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:47.374855 24542 raft_consensus.cc:348] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.375592 24542 raft_consensus.cc:374] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:47.375731 24542 raft_consensus.cc:729] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a34aa898db3492f8af58b49a84d4f15, State: Initialized, Role: FOLLOWER
I20260807 11:21:47.376547 24542 consensus_queue.cc:260] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.378413 24387 heartbeater.cc:503] Master 127.2.223.190:36435 was elected leader, sending a full tablet report...
I20260807 11:21:47.379037 24545 tablet_bootstrap.cc:492] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: No bootstrap required, opened a new log
I20260807 11:21:47.379304 24542 ts_tablet_manager.cc:1428] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Time spent starting tablet: real 0.018s user 0.006s sys 0.009s
I20260807 11:21:47.379347 24545 ts_tablet_manager.cc:1397] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260807 11:21:47.391356 24545 raft_consensus.cc:348] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.391976 24545 raft_consensus.cc:374] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:47.392097 24545 raft_consensus.cc:729] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6c9b8f3c7547cd8f79e70c5112f69a, State: Initialized, Role: FOLLOWER
I20260807 11:21:47.392906 24545 consensus_queue.cc:260] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.395218 24545 ts_tablet_manager.cc:1428] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260807 11:21:47.483413 24523 tablet.cc:2369] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:21:47.582713 24250 tablet.cc:2369] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:47.586199 24549 raft_consensus.cc:482] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:21:47.586596 24549 raft_consensus.cc:504] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.589866 24549 leader_election.cc:288] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a5a92cc335458c9a0b4adde1d66a6b (127.2.223.131:45577), 9a34aa898db3492f8af58b49a84d4f15 (127.2.223.130:36643)
I20260807 11:21:47.598097 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.589821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:47369 (local address 127.2.223.131:45577)
0807 11:21:47.590053 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.590061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.590093 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:21:47.590452 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.590458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.590585 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.590715 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:47.591290 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.592270 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.595214 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.596663 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.596887 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.596965 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.597617 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.597790 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:21:47.598093 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.589246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.223.129:53245 (local address 127.2.223.130:36643)
0807 11:21:47.589634 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.589641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.590652 (+ 1011us) server_negotiation.cc:407] Connection header received
0807 11:21:47.590873 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.590878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.590955 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.591074 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:21:47.591807 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.592748 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.595356 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.596750 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.596904 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.596965 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.597623 (+ 658us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.597790 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260807 11:21:47.598296 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.588941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.131:45577 (local address 127.2.223.129:47369)
0807 11:21:47.589911 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.589942 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.590170 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.590756 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.590761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.590789 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:47.591126 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.591136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.592421 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.592425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.594925 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.594948 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.597052 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.597059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.597342 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.597428 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.597542 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.597800 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":189,"thread_start_us":1170,"threads_started":1}
I20260807 11:21:47.598646 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.588941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.130:36643 (local address 127.2.223.129:53245)
0807 11:21:47.590520 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0807 11:21:47.590542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:21:47.590706 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:21:47.591107 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:21:47.591112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:21:47.591136 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:21:47.591609 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.591625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.592928 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.592934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.595132 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:21:47.595147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.597001 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.597009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:21:47.597357 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.597428 (+ 71us) client_negotiation.cc:769] Sending connection context
0807 11:21:47.597543 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 11:21:47.597800 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"mutex_wait_us":149,"thread_start_us":511,"threads_started":1}
I20260807 11:21:47.599725 24338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1786304873014a7da3bc0535b2c55d53" candidate_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a34aa898db3492f8af58b49a84d4f15" is_pre_election: true
I20260807 11:21:47.599700 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1786304873014a7da3bc0535b2c55d53" candidate_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" is_pre_election: true
I20260807 11:21:47.600342 24338 raft_consensus.cc:2454] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe6c9b8f3c7547cd8f79e70c5112f69a in term 0.
I20260807 11:21:47.600350 24473 raft_consensus.cc:2454] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe6c9b8f3c7547cd8f79e70c5112f69a in term 0.
I20260807 11:21:47.601200 24131 leader_election.cc:302] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a5a92cc335458c9a0b4adde1d66a6b, fe6c9b8f3c7547cd8f79e70c5112f69a; no voters:
I20260807 11:21:47.601823 24549 raft_consensus.cc:2781] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:21:47.602010 24549 raft_consensus.cc:482] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:21:47.602159 24549 raft_consensus.cc:3037] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:47.605660 24549 raft_consensus.cc:504] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.606751 24549 leader_election.cc:288] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [CANDIDATE]: Term 1 election: Requested vote from peers 95a5a92cc335458c9a0b4adde1d66a6b (127.2.223.131:45577), 9a34aa898db3492f8af58b49a84d4f15 (127.2.223.130:36643)
I20260807 11:21:47.607378 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1786304873014a7da3bc0535b2c55d53" candidate_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a5a92cc335458c9a0b4adde1d66a6b"
I20260807 11:21:47.607440 24338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1786304873014a7da3bc0535b2c55d53" candidate_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a34aa898db3492f8af58b49a84d4f15"
I20260807 11:21:47.607676 24473 raft_consensus.cc:3037] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:47.607674 24338 raft_consensus.cc:3037] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:47.611423 24473 raft_consensus.cc:2454] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe6c9b8f3c7547cd8f79e70c5112f69a in term 1.
I20260807 11:21:47.611423 24338 raft_consensus.cc:2454] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe6c9b8f3c7547cd8f79e70c5112f69a in term 1.
I20260807 11:21:47.612116 24131 leader_election.cc:302] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 95a5a92cc335458c9a0b4adde1d66a6b, 9a34aa898db3492f8af58b49a84d4f15, fe6c9b8f3c7547cd8f79e70c5112f69a; no voters:
I20260807 11:21:47.612620 24549 raft_consensus.cc:2781] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:47.613291 24549 raft_consensus.cc:686] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 1 LEADER]: Becoming Leader. State: Replica: fe6c9b8f3c7547cd8f79e70c5112f69a, State: Running, Role: LEADER
I20260807 11:21:47.613900 24549 consensus_queue.cc:237] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } }
I20260807 11:21:47.620359 24049 catalog_manager.cc:5495] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a reported cstate change: term changed from 0 to 1, leader changed from <none> to fe6c9b8f3c7547cd8f79e70c5112f69a (127.2.223.129). New cstate: current_term: 1 leader_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe6c9b8f3c7547cd8f79e70c5112f69a" member_type: VOTER last_known_addr { host: "127.2.223.129" port: 41119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 } health_report { overall_health: UNKNOWN } } }
W20260807 11:21:47.627992 24388 tablet.cc:2369] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:21:47.708155 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.701232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.2.223.129:41119)
0807 11:21:47.701438 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.701442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.701463 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:21:47.701742 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.701748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.701862 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.702001 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:21:47.702553 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.703259 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.705465 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.705943 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.706058 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.706826 (+ 768us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:21:47.707453 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:21:47.707610 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.707777 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.707861 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:21:47.719453 24473 raft_consensus.cc:1261] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 1 FOLLOWER]: Refusing update from remote peer fe6c9b8f3c7547cd8f79e70c5112f69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:47.719610 24338 raft_consensus.cc:1261] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 1 FOLLOWER]: Refusing update from remote peer fe6c9b8f3c7547cd8f79e70c5112f69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:21:47.720765 24552 consensus_queue.cc:1035] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a5a92cc335458c9a0b4adde1d66a6b" member_type: VOTER last_known_addr { host: "127.2.223.131" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:47.721282 24549 consensus_queue.cc:1035] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a34aa898db3492f8af58b49a84d4f15" member_type: VOTER last_known_addr { host: "127.2.223.130" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:21:47.739024 24555 mvcc.cc:205] Tried to move back new op lower bound from 7315872594805891072 to 7315872594392174592. Current Snapshot: MvccSnapshot[applied={T|T < 7315872594805891072}]
I20260807 11:21:47.760505 24048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:47.760895 24048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:21:47.765799 24048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 409e9e1bef7447e383124a80ea3c25bf: Sending DeleteTablet for 3 replicas of tablet 1786304873014a7da3bc0535b2c55d53
I20260807 11:21:47.767361 24453 tablet_service.cc:1514] Processing DeleteTablet for tablet 1786304873014a7da3bc0535b2c55d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:47 UTC) from {username='slave'} at 127.0.0.1:51494
I20260807 11:21:47.768028 24318 tablet_service.cc:1514] Processing DeleteTablet for tablet 1786304873014a7da3bc0535b2c55d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:47 UTC) from {username='slave'} at 127.0.0.1:50696
I20260807 11:21:47.768491 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1786304873014a7da3bc0535b2c55d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:21:47 UTC) from {username='slave'} at 127.0.0.1:52158
I20260807 11:21:47.769136 24566 tablet_replica.cc:330] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: stopping tablet replica
I20260807 11:21:47.769749 24566 raft_consensus.cc:2229] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:47.770421 24566 raft_consensus.cc:2258] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:47.770907 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24110
I20260807 11:21:47.773406 24566 ts_tablet_manager.cc:1905] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:47.783123 24567 tablet_replica.cc:330] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: stopping tablet replica
I20260807 11:21:47.783691 24566 ts_tablet_manager.cc:1918] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:47.783761 24567 raft_consensus.cc:2229] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:21:47.783931 24566 log.cc:1196] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-1/wal/wals/1786304873014a7da3bc0535b2c55d53
I20260807 11:21:47.784569 24567 raft_consensus.cc:2258] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:47.784716 24566 ts_tablet_manager.cc:1939] T 1786304873014a7da3bc0535b2c55d53 P 9a34aa898db3492f8af58b49a84d4f15: Deleting consensus metadata
I20260807 11:21:47.787561 24035 catalog_manager.cc:4841] TS 9a34aa898db3492f8af58b49a84d4f15 (127.2.223.130:36643): tablet 1786304873014a7da3bc0535b2c55d53 (table test-table [id=c215c416959c4cf2aa6383813437663d]) successfully deleted
I20260807 11:21:47.789661 24567 ts_tablet_manager.cc:1905] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:47.798094 24565 tablet_replica.cc:330] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: stopping tablet replica
I20260807 11:21:47.802153 24567 ts_tablet_manager.cc:1918] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:47.802446 24567 log.cc:1196] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-0/wal/wals/1786304873014a7da3bc0535b2c55d53
I20260807 11:21:47.803151 24567 ts_tablet_manager.cc:1939] T 1786304873014a7da3bc0535b2c55d53 P fe6c9b8f3c7547cd8f79e70c5112f69a: Deleting consensus metadata
I20260807 11:21:47.805627 24033 catalog_manager.cc:4841] TS fe6c9b8f3c7547cd8f79e70c5112f69a (127.2.223.129:41119): tablet 1786304873014a7da3bc0535b2c55d53 (table test-table [id=c215c416959c4cf2aa6383813437663d]) successfully deleted
I20260807 11:21:47.798765 24565 raft_consensus.cc:2229] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:21:47.806797 24565 raft_consensus.cc:2258] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:21:47.809984 24565 ts_tablet_manager.cc:1905] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:21:47.814015 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.773083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.2.223.129:41119)
0807 11:21:47.773308 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.773315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.783236 (+ 9921us) server_negotiation.cc:407] Connection header received
0807 11:21:47.783566 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.783575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.783722 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.783847 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:47.784738 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.786017 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.806114 (+ 20097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.806913 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:47.807089 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:47.811251 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:47.811270 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:47.811290 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:47.812296 (+ 1006us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:47.812380 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:47.812417 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:47.812563 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:47.813014 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 11:21:47.813197 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 11:21:47.816035 24244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:47.818756 24565 ts_tablet_manager.cc:1918] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:21:47.819012 24565 log.cc:1196] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Deleting WAL directory at /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786101369622807-2942-0/minicluster-data/ts-2/wal/wals/1786304873014a7da3bc0535b2c55d53
I20260807 11:21:47.819648 24565 ts_tablet_manager.cc:1939] T 1786304873014a7da3bc0535b2c55d53 P 95a5a92cc335458c9a0b4adde1d66a6b: Deleting consensus metadata
I20260807 11:21:47.821516 24034 catalog_manager.cc:4841] TS 95a5a92cc335458c9a0b4adde1d66a6b (127.2.223.131:45577): tablet 1786304873014a7da3bc0535b2c55d53 (table test-table [id=c215c416959c4cf2aa6383813437663d]) successfully deleted
I20260807 11:21:47.970280 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24110
I20260807 11:21:47.995971 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24253
I20260807 11:21:48.004514 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:47.996852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.2.223.130:36643)
0807 11:21:47.997124 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:47.997131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:47.997156 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:21:47.997273 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:47.997279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:47.997406 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:47.997535 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:47.998103 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:47.999405 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:48.000708 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:48.001409 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:48.001553 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:48.002899 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:48.002909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:48.002927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:48.003637 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:48.003695 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:48.003717 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:48.003875 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:48.004005 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 11:21:48.004147 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260807 11:21:48.005303 24381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:48.108518 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24253
I20260807 11:21:48.139617 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24391
I20260807 11:21:48.149014 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:48.140595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.2.223.131:45577)
0807 11:21:48.141144 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:48.141149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:48.141166 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:21:48.141226 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:48.141229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:48.141333 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:48.141468 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:48.141940 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:48.142764 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:48.144313 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:48.144841 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:48.145038 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:48.146655 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:48.146677 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:48.146696 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:48.147379 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:48.147480 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:48.147527 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:48.147691 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:48.148337 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 11:21:48.148506 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":300,"threads_started":1}
I20260807 11:21:48.150208 24517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:48.297497 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24391
I20260807 11:21:48.321238 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24016
I20260807 11:21:48.322724 24081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:48.439260 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24016
2026-08-07T11:21:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3615 ms)
[----------] 6 tests from EncryptionPolicyTest (25634 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:48 dist-test-slave-5l3k krb5kdc[24576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:48 dist-test-slave-5l3k krb5kdc[24576](info): set up 2 sockets
Aug 07 11:21:48 dist-test-slave-5l3k krb5kdc[24576](info): commencing operation
krb5kdc: starting...
W20260807 11:21:49.734818 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:50 dist-test-slave-5l3k krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:21:50.219954 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:50.677640 24589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:50.678015 24589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:50.678210 24589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:50.686731 24589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:50.686882 24589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:50.686990 24589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:50.687103 24589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:50.687168 24589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:50.698390 24589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:50.700130 24589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:50.701669 24589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:50.716817 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:50.716948 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:50.717399 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:50.718351 24589 server_base.cc:1034] running on GCE node
I20260807 11:21:50.719353 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:50.721386 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:50.722676 24589 hybrid_clock.cc:648] HybridClock initialized: now 1786101710722621 us; error 52 us; skew 500 ppm
I20260807 11:21:50.723127 24589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:50 dist-test-slave-5l3k krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:50.731709 24589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:21:50.733773 24589 webserver.cc:460] Webserver started at http://127.0.0.1:44111/ using document root <none> and password file <none>
I20260807 11:21:50.734750 24589 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:50.734925 24589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:50.735306 24589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:50.739080 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "231ca6c01b844b39a6cf422c41523f07"
format_stamp: "Formatted at 2026-08-07 11:21:50 on dist-test-slave-5l3k"
server_key: "5192f97c4e695067691de159ba6b7456"
server_key_iv: "97dec351692220cb7ce1e4758d5ad374"
server_key_version: "encryptionkey@0"
I20260807 11:21:50.740147 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "231ca6c01b844b39a6cf422c41523f07"
format_stamp: "Formatted at 2026-08-07 11:21:50 on dist-test-slave-5l3k"
server_key: "5192f97c4e695067691de159ba6b7456"
server_key_iv: "97dec351692220cb7ce1e4758d5ad374"
server_key_version: "encryptionkey@0"
I20260807 11:21:50.747236 24589 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 11:21:50.753566 24606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:50.755841 24589 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260807 11:21:50.756116 24589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "231ca6c01b844b39a6cf422c41523f07"
format_stamp: "Formatted at 2026-08-07 11:21:50 on dist-test-slave-5l3k"
server_key: "5192f97c4e695067691de159ba6b7456"
server_key_iv: "97dec351692220cb7ce1e4758d5ad374"
server_key_version: "encryptionkey@0"
I20260807 11:21:50.756413 24589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:50.791870 24589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:50.797983 24589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:50.798389 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:50.821641 24589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33561
I20260807 11:21:50.821684 24662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33561 every 8 connection(s)
I20260807 11:21:50.823432 24589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:50.829020 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:50.833024 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24589
I20260807 11:21:50.833315 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:50.834188 2942 external_mini_cluster.cc:1428] Setting key 7bb8d35664437a4d4337cb7390415e7c
I20260807 11:21:50.847447 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07: Bootstrap starting.
I20260807 11:21:50.853256 24663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:50.855144 24663 log.cc:824] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:50.860098 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07: No bootstrap required, opened a new log
Aug 07 11:21:50 dist-test-slave-5l3k krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:21:50.872053 24663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } }
I20260807 11:21:50.872711 24663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:50.872861 24663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231ca6c01b844b39a6cf422c41523f07, State: Initialized, Role: FOLLOWER
I20260807 11:21:50.873773 24663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } }
I20260807 11:21:50.874112 24663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:50.874313 24663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:50.874605 24663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:50.878005 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:50.837082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40471 (local address 127.0.0.1:33561)
0807 11:21:50.842281 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:50.842346 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:50.842417 (+ 71us) server_negotiation.cc:407] Connection header received
0807 11:21:50.845276 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:50.845322 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:50.846052 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:50.846671 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:50.849081 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:50.851903 (+ 2822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:50.854699 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:50.856259 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:50.856522 (+ 263us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:50.864762 (+ 8240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:50.864779 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:50.864818 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:50.864895 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:50.870659 (+ 5764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:50.872198 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:50.872209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:50.872219 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:50.872502 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:50.873327 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:50.873336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:50.873342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:50.874318 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:50.874850 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:50.875540 (+ 690us) server_negotiation.cc:299] Negotiation successful
0807 11:21:50.876101 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4814,"thread_start_us":4470,"threads_started":1}
I20260807 11:21:50.878013 24663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } }
I20260807 11:21:50.878952 24663 leader_election.cc:302] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231ca6c01b844b39a6cf422c41523f07; no voters:
I20260807 11:21:50.879767 24663 leader_election.cc:288] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:50.879875 24668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:50.880723 24668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [term 1 LEADER]: Becoming Leader. State: Replica: 231ca6c01b844b39a6cf422c41523f07, State: Running, Role: LEADER
I20260807 11:21:50.881578 24668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } }
I20260807 11:21:50.884183 24663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:50.884720 24670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231ca6c01b844b39a6cf422c41523f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } } }
I20260807 11:21:50.885104 24670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:50.885908 24669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231ca6c01b844b39a6cf422c41523f07. Latest consensus state: current_term: 1 leader_uuid: "231ca6c01b844b39a6cf422c41523f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231ca6c01b844b39a6cf422c41523f07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33561 } } }
I20260807 11:21:50.886189 24672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:50.886250 24669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:50.890293 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:21:50.892473 24672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:50.918605 24683 mvcc.cc:205] Tried to move back new op lower bound from 7315872607887994880 to 7315872607773339648. Current Snapshot: MvccSnapshot[applied={T|T < 7315872607887994880}]
I20260807 11:21:50.921789 24672 catalog_manager.cc:1347] Generated new cluster ID: 20bf4fa925f54c68b283145c66fceb73
I20260807 11:21:50.922013 24672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:50.959599 24672 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:50.962518 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:50.904944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41897 (local address 127.0.0.1:33561)
0807 11:21:50.905207 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:50.905214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:50.905245 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:21:50.905331 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:50.905337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:50.905423 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:50.905581 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:50.908003 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:50.910827 (+ 2824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:50.912987 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:50.913751 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:50.913869 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:50.926216 (+ 12347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:50.926228 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:50.926237 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:50.926303 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:50.943922 (+ 17619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:50.952840 (+ 8918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:50.952852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:50.952858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:50.953136 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:50.959825 (+ 6689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:50.959835 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:50.959843 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:50.960403 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:50.960959 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:50.961533 (+ 574us) server_negotiation.cc:299] Negotiation successful
0807 11:21:50.961894 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":123}
I20260807 11:21:50.963872 24672 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:50.982425 24672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231ca6c01b844b39a6cf422c41523f07: Generated new TSK 0
I20260807 11:21:50.983829 24672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:50.993227 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24589
I20260807 11:21:50.995795 24658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:51.106559 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24589
W20260807 11:21:51.137270 24574 connection.cc:535] client connection to 127.0.0.1:33561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33561): Connection reset by peer (error 104)
2026-08-07T11:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2652 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:51 dist-test-slave-5l3k krb5kdc[24695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:51 dist-test-slave-5l3k krb5kdc[24695](info): set up 2 sockets
Aug 07 11:21:51 dist-test-slave-5l3k krb5kdc[24695](info): commencing operation
krb5kdc: starting...
W20260807 11:21:52.417025 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.004s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:52 dist-test-slave-5l3k krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:21:52.819979 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:53.205641 24708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:53.206036 24708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:53.206223 24708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:53.213943 24708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:53.214100 24708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:53.214226 24708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:53.214340 24708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:53.214428 24708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:53.225421 24708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:53.227051 24708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:53.228617 24708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:53.242331 24714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:53.242336 24716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:53.242270 24713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:53.248405 24708 server_base.cc:1034] running on GCE node
I20260807 11:21:53.249455 24708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:53.251272 24708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:53.252540 24708 hybrid_clock.cc:648] HybridClock initialized: now 1786101713252496 us; error 47 us; skew 500 ppm
I20260807 11:21:53.252948 24708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:53 dist-test-slave-5l3k krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:53.256596 24708 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:21:53.258610 24708 webserver.cc:460] Webserver started at http://127.0.0.1:43999/ using document root <none> and password file <none>
I20260807 11:21:53.259490 24708 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:53.259626 24708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:53.260022 24708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:53.263619 24708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "79c7df7347b24713b178e2a6fb2aed73"
format_stamp: "Formatted at 2026-08-07 11:21:53 on dist-test-slave-5l3k"
server_key: "7d83a24c532eb234e1fd884f472dd87a"
server_key_iv: "86487c7acc1f561ed751c074ad72db5c"
server_key_version: "encryptionkey@0"
I20260807 11:21:53.264547 24708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "79c7df7347b24713b178e2a6fb2aed73"
format_stamp: "Formatted at 2026-08-07 11:21:53 on dist-test-slave-5l3k"
server_key: "7d83a24c532eb234e1fd884f472dd87a"
server_key_iv: "86487c7acc1f561ed751c074ad72db5c"
server_key_version: "encryptionkey@0"
I20260807 11:21:53.270828 24708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 11:21:53.275003 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:53.276647 24708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:21:53.276885 24708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "79c7df7347b24713b178e2a6fb2aed73"
format_stamp: "Formatted at 2026-08-07 11:21:53 on dist-test-slave-5l3k"
server_key: "7d83a24c532eb234e1fd884f472dd87a"
server_key_iv: "86487c7acc1f561ed751c074ad72db5c"
server_key_version: "encryptionkey@0"
I20260807 11:21:53.277141 24708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:53.295394 24708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:53.300839 24708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:53.301194 24708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:53.322137 24708 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41729
I20260807 11:21:53.322151 24776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41729 every 8 connection(s)
I20260807 11:21:53.323884 24708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:53.328433 24778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:53.332710 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24708
I20260807 11:21:53.332934 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:53.333425 2942 external_mini_cluster.cc:1428] Setting key 57a988667904981ecbd7a2656d07f250
I20260807 11:21:53.343386 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73: Bootstrap starting.
I20260807 11:21:53.348132 24778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:53.349567 24778 log.cc:824] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:53.350239 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:53.335619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57427 (local address 127.0.0.1:41729)
0807 11:21:53.336586 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:53.336600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:53.336655 (+ 55us) server_negotiation.cc:407] Connection header received
0807 11:21:53.338671 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:53.338709 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:53.339305 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:53.339729 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:53.340539 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:53.341997 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:53.344069 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:53.345011 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:53.345194 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:53.347311 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:53.347321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:53.347339 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:53.347980 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:53.348080 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:53.348111 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:53.348310 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:53.348716 (+ 406us) server_negotiation.cc:299] Negotiation successful
0807 11:21:53.348947 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":457,"threads_started":1}
I20260807 11:21:53.353488 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73: No bootstrap required, opened a new log
I20260807 11:21:53.355289 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:21:53.363309 24778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260807 11:21:53.363951 24778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:53.364104 24778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c7df7347b24713b178e2a6fb2aed73, State: Initialized, Role: FOLLOWER
I20260807 11:21:53.364951 24778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260807 11:21:53.365291 24778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:53.365471 24778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:53.365734 24778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:53.368794 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:53.360702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52665 (local address 127.0.0.1:41729)
0807 11:21:53.360967 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:53.360975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:53.361385 (+ 410us) server_negotiation.cc:407] Connection header received
0807 11:21:53.361611 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:53.361616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:53.361708 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:53.361807 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:53.362606 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:53.363420 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:53.365500 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:53.366081 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:53.366197 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:53.367794 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:53.367799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:53.367803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:53.367827 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:53.367907 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:53.367926 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:53.368051 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:53.368333 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 11:21:53.368433 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260807 11:21:53.369450 24778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260807 11:21:53.370105 24778 leader_election.cc:302] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c7df7347b24713b178e2a6fb2aed73; no voters:
I20260807 11:21:53.370926 24778 leader_election.cc:288] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:53.371057 24789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:53.371788 24789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [term 1 LEADER]: Becoming Leader. State: Replica: 79c7df7347b24713b178e2a6fb2aed73, State: Running, Role: LEADER
I20260807 11:21:53.372535 24789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } }
I20260807 11:21:53.373656 24778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:53.376260 24791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c7df7347b24713b178e2a6fb2aed73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } } }
I20260807 11:21:53.376303 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c7df7347b24713b178e2a6fb2aed73. Latest consensus state: current_term: 1 leader_uuid: "79c7df7347b24713b178e2a6fb2aed73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c7df7347b24713b178e2a6fb2aed73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41729 } } }
I20260807 11:21:53.376726 24791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:53.376739 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:53.380172 24797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:53.385454 24797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:53.395814 24797 catalog_manager.cc:1347] Generated new cluster ID: 5f47b8c8779f415dbd4fefd925eee558
I20260807 11:21:53.395961 24797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:53.409587 24797 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:53.411203 24797 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:53.418884 24797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c7df7347b24713b178e2a6fb2aed73: Generated new TSK 0
I20260807 11:21:53.420200 24797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:53.435806 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24708
W20260807 11:21:53.437242 24772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57427
W20260807 11:21:53.438300 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:21:53.438548 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24708
W20260807 11:21:53.461575 24693 connection.cc:535] client connection to 127.0.0.1:41729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41729): Connection reset by peer (error 104)
2026-08-07T11:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:53 dist-test-slave-5l3k krb5kdc[24812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:53 dist-test-slave-5l3k krb5kdc[24812](info): set up 2 sockets
Aug 07 11:21:53 dist-test-slave-5l3k krb5kdc[24812](info): commencing operation
krb5kdc: starting...
W20260807 11:21:54.649705 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.009s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:54 dist-test-slave-5l3k krb5kdc[24812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:21:55.052130 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:55.423802 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:55.424141 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:55.424247 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:55.431813 24825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:55.431942 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:55.431990 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:55.432060 24825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:55.432101 24825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:55.442373 24825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:55.443737 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:55.445076 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:55.457692 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:55.457609 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:55.457785 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:55.458999 24825 server_base.cc:1034] running on GCE node
I20260807 11:21:55.459962 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:55.461630 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:55.462886 24825 hybrid_clock.cc:648] HybridClock initialized: now 1786101715462837 us; error 47 us; skew 500 ppm
I20260807 11:21:55.463291 24825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:55 dist-test-slave-5l3k krb5kdc[24812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:55.466919 24825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:21:55.468668 24825 webserver.cc:460] Webserver started at http://127.0.0.1:44717/ using document root <none> and password file <none>
I20260807 11:21:55.469458 24825 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:55.469587 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:55.469897 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:55.473486 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "9e54a7a53339438f9a6bfea49f2ab9e3"
format_stamp: "Formatted at 2026-08-07 11:21:55 on dist-test-slave-5l3k"
server_key: "7425578043ff10a8d1ff1ee6f6c8823f"
server_key_iv: "7cd180a9ec7730cf65d6a84a599b48c6"
server_key_version: "encryptionkey@0"
I20260807 11:21:55.474606 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "9e54a7a53339438f9a6bfea49f2ab9e3"
format_stamp: "Formatted at 2026-08-07 11:21:55 on dist-test-slave-5l3k"
server_key: "7425578043ff10a8d1ff1ee6f6c8823f"
server_key_iv: "7cd180a9ec7730cf65d6a84a599b48c6"
server_key_version: "encryptionkey@0"
I20260807 11:21:55.481029 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:55.485216 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:55.486871 24825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:21:55.487093 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "9e54a7a53339438f9a6bfea49f2ab9e3"
format_stamp: "Formatted at 2026-08-07 11:21:55 on dist-test-slave-5l3k"
server_key: "7425578043ff10a8d1ff1ee6f6c8823f"
server_key_iv: "7cd180a9ec7730cf65d6a84a599b48c6"
server_key_version: "encryptionkey@0"
I20260807 11:21:55.487342 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:55.493711 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:55.498458 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:55.498859 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:55.518528 24825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44063
I20260807 11:21:55.518594 24895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44063 every 8 connection(s)
I20260807 11:21:55.520147 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:55.521416 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24825
I20260807 11:21:55.521636 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:55.522194 2942 external_mini_cluster.cc:1428] Setting key 5e0f7daa69d53a82fbd534ccdce2a815
I20260807 11:21:55.525446 24896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:55.532267 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:55.524235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53757 (local address 127.0.0.1:44063)
0807 11:21:55.525124 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:55.525140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:55.525198 (+ 58us) server_negotiation.cc:407] Connection header received
0807 11:21:55.527721 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:55.527758 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:55.528312 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:55.528843 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:55.529991 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:55.530001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:55.530020 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:55.530046 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:55.530145 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:55.530175 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:55.530438 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:55.530620 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 11:21:55.530851 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":343,"threads_started":1}
I20260807 11:21:55.536381 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:21:55.540874 24896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3: Bootstrap starting.
I20260807 11:21:55.545053 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:55.542239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49057 (local address 127.0.0.1:44063)
0807 11:21:55.542426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:55.542432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:55.542958 (+ 526us) server_negotiation.cc:407] Connection header received
0807 11:21:55.543121 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:55.543128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:55.543208 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:55.543348 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:55.544340 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:55.544346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:55.544351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:21:55.544370 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:55.544421 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:21:55.544441 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:55.544563 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:55.544659 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 11:21:55.544726 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 11:21:55.545205 24896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:55.546473 24896 log.cc:824] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:55.549572 24896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3: No bootstrap required, opened a new log
I20260807 11:21:55.557514 24896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } }
I20260807 11:21:55.558130 24896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:55.558300 24896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e54a7a53339438f9a6bfea49f2ab9e3, State: Initialized, Role: FOLLOWER
I20260807 11:21:55.559130 24896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } }
I20260807 11:21:55.559427 24896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:55.559619 24896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:55.559834 24896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:55.562338 24896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } }
I20260807 11:21:55.563025 24896 leader_election.cc:302] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e54a7a53339438f9a6bfea49f2ab9e3; no voters:
I20260807 11:21:55.563750 24896 leader_election.cc:288] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:55.563972 24908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:55.564674 24908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e54a7a53339438f9a6bfea49f2ab9e3, State: Running, Role: LEADER
I20260807 11:21:55.565320 24908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } }
I20260807 11:21:55.566390 24896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:55.568912 24910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e54a7a53339438f9a6bfea49f2ab9e3. Latest consensus state: current_term: 1 leader_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } } }
I20260807 11:21:55.569458 24910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:55.569324 24909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e54a7a53339438f9a6bfea49f2ab9e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44063 } } }
I20260807 11:21:55.569619 24909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:55.571550 24916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:55.577296 24916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:55.589057 24916 catalog_manager.cc:1347] Generated new cluster ID: d78df3e688a1496d8fb48f4de3a36f5d
I20260807 11:21:55.589265 24916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:55.609875 24916 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:55.610526 24916 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:55.623848 24916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e54a7a53339438f9a6bfea49f2ab9e3: Generated new TSK 0
I20260807 11:21:55.625097 24916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:55.635689 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24825
W20260807 11:21:55.637027 24891 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53757
W20260807 11:21:55.637907 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:21:55.638091 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24825
2026-08-07T11:21:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:55 dist-test-slave-5l3k krb5kdc[24929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:55 dist-test-slave-5l3k krb5kdc[24929](info): set up 2 sockets
Aug 07 11:21:55 dist-test-slave-5l3k krb5kdc[24929](info): commencing operation
krb5kdc: starting...
W20260807 11:21:56.837134 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:57 dist-test-slave-5l3k krb5kdc[24929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:21:57.238917 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:57.620422 24944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:57.620812 24944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:57.620995 24944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:57.628713 24944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:57.628901 24944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:57.629024 24944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:57.629146 24944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:57.629227 24944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:57.640858 24944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:57.642567 24944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:57.644114 24944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:21:57.657284 24949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:57.657488 24950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:21:57.657635 24953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:21:57.658855 24944 server_base.cc:1034] running on GCE node
I20260807 11:21:57.659904 24944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:21:57.661712 24944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:21:57.663036 24944 hybrid_clock.cc:648] HybridClock initialized: now 1786101717662840 us; error 170 us; skew 500 ppm
I20260807 11:21:57.663496 24944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:21:57 dist-test-slave-5l3k krb5kdc[24929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:21:57.667248 24944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:21:57.669162 24944 webserver.cc:460] Webserver started at http://127.0.0.1:39991/ using document root <none> and password file <none>
I20260807 11:21:57.670044 24944 fs_manager.cc:359] Metadata directory not provided
I20260807 11:21:57.670220 24944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:21:57.670641 24944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:21:57.674080 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "d4f3c486b0f043698905cb3ab2ffc922"
format_stamp: "Formatted at 2026-08-07 11:21:57 on dist-test-slave-5l3k"
server_key: "c733a9f5ff8f87111138cf50228b30e8"
server_key_iv: "567c4384b399f8be686d16310bb1b7a1"
server_key_version: "encryptionkey@0"
I20260807 11:21:57.675019 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "d4f3c486b0f043698905cb3ab2ffc922"
format_stamp: "Formatted at 2026-08-07 11:21:57 on dist-test-slave-5l3k"
server_key: "c733a9f5ff8f87111138cf50228b30e8"
server_key_iv: "567c4384b399f8be686d16310bb1b7a1"
server_key_version: "encryptionkey@0"
I20260807 11:21:57.681267 24944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:21:57.685542 24960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:21:57.687512 24944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:21:57.687774 24944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "d4f3c486b0f043698905cb3ab2ffc922"
format_stamp: "Formatted at 2026-08-07 11:21:57 on dist-test-slave-5l3k"
server_key: "c733a9f5ff8f87111138cf50228b30e8"
server_key_iv: "567c4384b399f8be686d16310bb1b7a1"
server_key_version: "encryptionkey@0"
I20260807 11:21:57.688048 24944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:21:57.715319 24944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:21:57.720631 24944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:21:57.721004 24944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:21:57.743701 24944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42723
I20260807 11:21:57.743729 25014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42723 every 8 connection(s)
I20260807 11:21:57.745442 24944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:21:57.749959 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 24944
I20260807 11:21:57.750134 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:21:57.750193 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:21:57.750804 2942 external_mini_cluster.cc:1428] Setting key ed1983dfd5a5ad3b3b12e57a08a11ac2
I20260807 11:21:57.764669 25015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922: Bootstrap starting.
Aug 07 11:21:57 dist-test-slave-5l3k krb5kdc[24929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:21:57.768702 25015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922: Neither blocks nor log segments found. Creating new log.
I20260807 11:21:57.770110 25015 log.cc:824] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922: Log is configured to *not* fsync() on all Append() calls
I20260807 11:21:57.773443 25015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922: No bootstrap required, opened a new log
I20260807 11:21:57.775938 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:57.752927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41839 (local address 127.0.0.1:42723)
0807 11:21:57.753858 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:57.753874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:57.753922 (+ 48us) server_negotiation.cc:407] Connection header received
0807 11:21:57.755829 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:57.755859 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:57.756372 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:57.756874 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:57.758610 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:57.759971 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:57.761355 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:57.762172 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:57.762406 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:57.767477 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:57.767487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:57.767505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:57.767554 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:57.771312 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:57.772205 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:57.772213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:57.772221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:57.772404 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:57.772851 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:57.772858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:57.772866 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:57.773415 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:57.773787 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:57.774268 (+ 481us) server_negotiation.cc:299] Negotiation successful
0807 11:21:57.774802 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":414,"threads_started":1}
I20260807 11:21:57.780189 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:21:57.782416 25015 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260807 11:21:57.783120 25015 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:21:57.783260 25015 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f3c486b0f043698905cb3ab2ffc922, State: Initialized, Role: FOLLOWER
I20260807 11:21:57.784006 25015 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260807 11:21:57.784282 25015 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:21:57.784456 25015 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:21:57.784633 25015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:21:57.788407 25015 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260807 11:21:57.789124 25015 leader_election.cc:302] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f3c486b0f043698905cb3ab2ffc922; no voters:
I20260807 11:21:57.789915 25015 leader_election.cc:288] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:21:57.790042 25025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:21:57.791489 25025 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [term 1 LEADER]: Becoming Leader. State: Replica: d4f3c486b0f043698905cb3ab2ffc922, State: Running, Role: LEADER
I20260807 11:21:57.792330 25025 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260807 11:21:57.793037 25015 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:21:57.796578 25027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f3c486b0f043698905cb3ab2ffc922. Latest consensus state: current_term: 1 leader_uuid: "d4f3c486b0f043698905cb3ab2ffc922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260807 11:21:57.797147 25027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:57.804051 25026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f3c486b0f043698905cb3ab2ffc922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3c486b0f043698905cb3ab2ffc922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260807 11:21:57.805248 25026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922 [sys.catalog]: This master's current role is: LEADER
I20260807 11:21:57.807730 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:21:57.785095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37639 (local address 127.0.0.1:42723)
0807 11:21:57.785327 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:21:57.785336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:21:57.785903 (+ 567us) server_negotiation.cc:407] Connection header received
0807 11:21:57.786135 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:21:57.786139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:21:57.786210 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:21:57.786295 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:21:57.788438 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:57.789251 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:57.791263 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:21:57.791986 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:21:57.792149 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:21:57.795768 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:21:57.795775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:21:57.795780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:21:57.795832 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:21:57.800614 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:57.802415 (+ 1801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:57.802427 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:57.802434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:57.802737 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:21:57.803581 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:21:57.803592 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:21:57.803599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:21:57.804231 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:21:57.804604 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0807 11:21:57.806686 (+ 2082us) server_negotiation.cc:299] Negotiation successful
0807 11:21:57.807106 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:21:57.812551 25031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:21:57.819135 25031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:21:57.831262 25031 catalog_manager.cc:1347] Generated new cluster ID: d2272e2c5f2a49dd806f902ba2db8f30
I20260807 11:21:57.831427 25031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:21:57.841899 25031 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:21:57.844062 25031 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:21:57.853052 25031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f3c486b0f043698905cb3ab2ffc922: Generated new TSK 0
I20260807 11:21:57.854373 25031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:21:57.861210 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 24944
I20260807 11:21:57.863577 25010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:21:57.957975 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 24944
2026-08-07T11:21:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:21:58 dist-test-slave-5l3k krb5kdc[25049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:21:58 dist-test-slave-5l3k krb5kdc[25049](info): set up 2 sockets
Aug 07 11:21:58 dist-test-slave-5l3k krb5kdc[25049](info): commencing operation
krb5kdc: starting...
W20260807 11:21:59.174265 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:21:59 dist-test-slave-5l3k krb5kdc[25049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:21:59.581722 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:21:59.974884 25063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:21:59.975342 25063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:21:59.975517 25063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:21:59.983692 25063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:21:59.983906 25063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:21:59.984036 25063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:21:59.984145 25063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:21:59.984237 25063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:21:59.995226 25063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:21:59.996824 25063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:21:59.998410 25063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:00.011973 25068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:00.011916 25071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:00.012005 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:00.013120 25063 server_base.cc:1034] running on GCE node
I20260807 11:22:00.014127 25063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:00.015847 25063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:00.017089 25063 hybrid_clock.cc:648] HybridClock initialized: now 1786101720017048 us; error 39 us; skew 500 ppm
I20260807 11:22:00.017494 25063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:00 dist-test-slave-5l3k krb5kdc[25049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:00.021090 25063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:00.022929 25063 webserver.cc:460] Webserver started at http://127.0.0.1:37931/ using document root <none> and password file <none>
I20260807 11:22:00.023732 25063 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:00.023849 25063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:00.024223 25063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:00.027697 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "599009acc7dd4cd9b3bc8f71948216aa"
format_stamp: "Formatted at 2026-08-07 11:22:00 on dist-test-slave-5l3k"
server_key: "a3077a2b4aa0558d8395f9c7a3ed9f8e"
server_key_iv: "e48b243b8e84c094549cbd88cc4502d7"
server_key_version: "encryptionkey@0"
I20260807 11:22:00.028597 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "599009acc7dd4cd9b3bc8f71948216aa"
format_stamp: "Formatted at 2026-08-07 11:22:00 on dist-test-slave-5l3k"
server_key: "a3077a2b4aa0558d8395f9c7a3ed9f8e"
server_key_iv: "e48b243b8e84c094549cbd88cc4502d7"
server_key_version: "encryptionkey@0"
I20260807 11:22:00.034817 25063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:22:00.038937 25078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:00.040640 25063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:22:00.040894 25063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "599009acc7dd4cd9b3bc8f71948216aa"
format_stamp: "Formatted at 2026-08-07 11:22:00 on dist-test-slave-5l3k"
server_key: "a3077a2b4aa0558d8395f9c7a3ed9f8e"
server_key_iv: "e48b243b8e84c094549cbd88cc4502d7"
server_key_version: "encryptionkey@0"
I20260807 11:22:00.041145 25063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:00.064352 25063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:00.069037 25063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:00.069388 25063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:00.091130 25063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37645
I20260807 11:22:00.091168 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37645 every 8 connection(s)
I20260807 11:22:00.092756 25063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:00.094518 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25063
I20260807 11:22:00.094740 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:00.095204 2942 external_mini_cluster.cc:1428] Setting key 892d5001608a7fa7a9bfd3ed89c7b5a4
I20260807 11:22:00.098344 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:00.109915 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:00.097273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60747 (local address 127.0.0.1:37645)
0807 11:22:00.098228 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:00.098241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:00.098287 (+ 46us) server_negotiation.cc:407] Connection header received
0807 11:22:00.100290 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:00.100317 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:00.100859 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:00.101311 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:00.101954 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:00.103423 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:00.104886 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:00.105626 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:00.105851 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:00.107277 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:00.107286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:00.107303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:00.107915 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:00.108010 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:00.108046 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:00.108200 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:00.108458 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 11:22:00.108783 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":434,"threads_started":1}
I20260807 11:22:00.112573 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa: Bootstrap starting.
I20260807 11:22:00.114154 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:00.116758 25134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:00.118283 25134 log.cc:824] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:00.122223 25134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa: No bootstrap required, opened a new log
I20260807 11:22:00.125636 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:00.119017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35637 (local address 127.0.0.1:37645)
0807 11:22:00.119377 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:00.119385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:00.119707 (+ 322us) server_negotiation.cc:407] Connection header received
0807 11:22:00.119949 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:00.119953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:00.120017 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:00.120127 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:00.120706 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:00.121446 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:00.122973 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:00.123450 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:00.123598 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:00.124863 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:00.124868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:00.124871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:00.124896 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:00.124936 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:00.124952 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:00.125063 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:00.125220 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 11:22:00.125325 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260807 11:22:00.132704 25134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } }
I20260807 11:22:00.133427 25134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:00.133585 25134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599009acc7dd4cd9b3bc8f71948216aa, State: Initialized, Role: FOLLOWER
I20260807 11:22:00.134464 25134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } }
I20260807 11:22:00.134858 25134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:00.135056 25134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:00.135289 25134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:00.138167 25134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } }
I20260807 11:22:00.138955 25134 leader_election.cc:302] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599009acc7dd4cd9b3bc8f71948216aa; no voters:
I20260807 11:22:00.139792 25134 leader_election.cc:288] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:00.139881 25144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:00.141762 25144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [term 1 LEADER]: Becoming Leader. State: Replica: 599009acc7dd4cd9b3bc8f71948216aa, State: Running, Role: LEADER
I20260807 11:22:00.142372 25144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } }
I20260807 11:22:00.142881 25134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:00.146087 25145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599009acc7dd4cd9b3bc8f71948216aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } } }
I20260807 11:22:00.146669 25145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:00.147810 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599009acc7dd4cd9b3bc8f71948216aa. Latest consensus state: current_term: 1 leader_uuid: "599009acc7dd4cd9b3bc8f71948216aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599009acc7dd4cd9b3bc8f71948216aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37645 } } }
I20260807 11:22:00.148289 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:00.149667 25151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:00.155786 25151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:00.168314 25151 catalog_manager.cc:1347] Generated new cluster ID: 53ad63d05ed04254bd897d520bde5dad
I20260807 11:22:00.168483 25151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:00.187079 25151 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:00.188563 25151 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:00.199376 25151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599009acc7dd4cd9b3bc8f71948216aa: Generated new TSK 0
I20260807 11:22:00.200700 25151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:00.216497 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25063
W20260807 11:22:00.217945 25129 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60747
W20260807 11:22:00.218823 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:00.219003 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25063
2026-08-07T11:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:00 dist-test-slave-5l3k krb5kdc[25165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:00 dist-test-slave-5l3k krb5kdc[25165](info): set up 2 sockets
Aug 07 11:22:00 dist-test-slave-5l3k krb5kdc[25165](info): commencing operation
krb5kdc: starting...
W20260807 11:22:01.425781 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.002s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:01 dist-test-slave-5l3k krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:01.830034 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:02.237279 25178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:02.237644 25178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:02.237852 25178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:02.245911 25178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:02.246174 25178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:02.246280 25178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:02.246400 25178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:02.246511 25178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:02.257457 25178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:02.259069 25178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:02.260648 25178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:02.274814 25183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:02.274888 25187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:02.274916 25184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:02.276064 25178 server_base.cc:1034] running on GCE node
I20260807 11:22:02.277109 25178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:02.278966 25178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:02.280249 25178 hybrid_clock.cc:648] HybridClock initialized: now 1786101722280201 us; error 57 us; skew 500 ppm
I20260807 11:22:02.280687 25178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:02 dist-test-slave-5l3k krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:02.284440 25178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:02.286217 25178 webserver.cc:460] Webserver started at http://127.0.0.1:36965/ using document root <none> and password file <none>
I20260807 11:22:02.287119 25178 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:02.287256 25178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:02.287564 25178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:02.291096 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "d417768092464edc97f85a0fe2939bfe"
format_stamp: "Formatted at 2026-08-07 11:22:02 on dist-test-slave-5l3k"
server_key: "3acc452b6e3dcb5480c7b27ea3d1c494"
server_key_iv: "c85f1af103aa9112b9f6714116ee315d"
server_key_version: "encryptionkey@0"
I20260807 11:22:02.291972 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "d417768092464edc97f85a0fe2939bfe"
format_stamp: "Formatted at 2026-08-07 11:22:02 on dist-test-slave-5l3k"
server_key: "3acc452b6e3dcb5480c7b27ea3d1c494"
server_key_iv: "c85f1af103aa9112b9f6714116ee315d"
server_key_version: "encryptionkey@0"
I20260807 11:22:02.298363 25178 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260807 11:22:02.302817 25194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:02.304606 25178 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260807 11:22:02.304811 25178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "d417768092464edc97f85a0fe2939bfe"
format_stamp: "Formatted at 2026-08-07 11:22:02 on dist-test-slave-5l3k"
server_key: "3acc452b6e3dcb5480c7b27ea3d1c494"
server_key_iv: "c85f1af103aa9112b9f6714116ee315d"
server_key_version: "encryptionkey@0"
I20260807 11:22:02.305025 25178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:02.312067 25178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:02.316964 25178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:02.317279 25178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:02.337597 25178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32955
I20260807 11:22:02.337644 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32955 every 8 connection(s)
I20260807 11:22:02.339313 25178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:02.343865 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25178
I20260807 11:22:02.344090 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:02.344116 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:02.344683 2942 external_mini_cluster.cc:1428] Setting key 10e66f014417e17eaaed985489fbeebe
I20260807 11:22:02.354130 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:02.346871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41759 (local address 127.0.0.1:32955)
0807 11:22:02.347841 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:02.347859 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:02.347909 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:22:02.349875 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:02.349904 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:02.350337 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:02.350942 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:02.352200 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:02.352207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:02.352223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:02.352238 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:02.352305 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:02.352326 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:02.352625 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:02.352789 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 11:22:02.353006 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":409,"threads_started":1}
I20260807 11:22:02.359002 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:02.360494 25249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe: Bootstrap starting.
I20260807 11:22:02.364962 25249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:02.366446 25249 log.cc:824] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:02.367036 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:02.364064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34489 (local address 127.0.0.1:32955)
0807 11:22:02.364270 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:02.364279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:02.364739 (+ 460us) server_negotiation.cc:407] Connection header received
0807 11:22:02.365003 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:02.365008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:02.365091 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:02.365202 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:02.366276 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:02.366282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:02.366286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:02.366313 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:02.366357 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:02.366372 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:02.366541 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:02.366673 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 11:22:02.366754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:22:02.370406 25249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe: No bootstrap required, opened a new log
I20260807 11:22:02.379285 25249 raft_consensus.cc:348] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260807 11:22:02.379814 25249 raft_consensus.cc:374] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:02.379941 25249 raft_consensus.cc:729] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d417768092464edc97f85a0fe2939bfe, State: Initialized, Role: FOLLOWER
I20260807 11:22:02.380785 25249 consensus_queue.cc:260] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260807 11:22:02.381150 25249 raft_consensus.cc:388] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:02.381361 25249 raft_consensus.cc:482] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:02.381557 25249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:02.384099 25249 raft_consensus.cc:504] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260807 11:22:02.384667 25249 leader_election.cc:302] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d417768092464edc97f85a0fe2939bfe; no voters:
I20260807 11:22:02.385277 25249 leader_election.cc:288] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:02.385486 25259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:02.386327 25259 raft_consensus.cc:686] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [term 1 LEADER]: Becoming Leader. State: Replica: d417768092464edc97f85a0fe2939bfe, State: Running, Role: LEADER
I20260807 11:22:02.387274 25259 consensus_queue.cc:237] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260807 11:22:02.388024 25249 sys_catalog.cc:564] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:02.390761 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d417768092464edc97f85a0fe2939bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } } }
I20260807 11:22:02.391615 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:02.391417 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d417768092464edc97f85a0fe2939bfe. Latest consensus state: current_term: 1 leader_uuid: "d417768092464edc97f85a0fe2939bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d417768092464edc97f85a0fe2939bfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } } }
I20260807 11:22:02.391840 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:02.393849 25269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:02.400772 25269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:02.412420 25269 catalog_manager.cc:1347] Generated new cluster ID: 9394c5a8d1e942b5a38b22b21d2a488e
I20260807 11:22:02.412580 25269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:02.437086 25269 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:02.437603 25269 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:02.449287 25269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d417768092464edc97f85a0fe2939bfe: Generated new TSK 0
I20260807 11:22:02.450275 25269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:02.463725 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25178
W20260807 11:22:02.465135 25244 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41759
W20260807 11:22:02.466027 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25178: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:02.466229 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25178
2026-08-07T11:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:02 dist-test-slave-5l3k krb5kdc[25281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:02 dist-test-slave-5l3k krb5kdc[25281](info): set up 2 sockets
Aug 07 11:22:02 dist-test-slave-5l3k krb5kdc[25281](info): commencing operation
krb5kdc: starting...
W20260807 11:22:03.680735 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:03 dist-test-slave-5l3k krb5kdc[25281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:04.087909 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:04.469316 25294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:04.469748 25294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:04.469936 25294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:04.477965 25294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:04.478179 25294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:04.478307 25294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:04.478422 25294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:04.478526 25294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:04.489437 25294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:04.491112 25294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:04.492658 25294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:04.506466 25302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:04.506587 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:04.506584 25300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:04.507572 25294 server_base.cc:1034] running on GCE node
I20260807 11:22:04.508579 25294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:04.510294 25294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:04.511565 25294 hybrid_clock.cc:648] HybridClock initialized: now 1786101724511512 us; error 75 us; skew 500 ppm
I20260807 11:22:04.512058 25294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:04 dist-test-slave-5l3k krb5kdc[25281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:04.515739 25294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:04.517576 25294 webserver.cc:460] Webserver started at http://127.0.0.1:36841/ using document root <none> and password file <none>
I20260807 11:22:04.518442 25294 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:04.518674 25294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:04.519058 25294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:04.522675 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "aa8a06dd381f4c3f9061aba141a7d4b2"
format_stamp: "Formatted at 2026-08-07 11:22:04 on dist-test-slave-5l3k"
server_key: "c8d8a907cc27b881c1308b644bf553c3"
server_key_iv: "071556e8b67dd4e2bcd2e7c39d5da18e"
server_key_version: "encryptionkey@0"
I20260807 11:22:04.523752 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "aa8a06dd381f4c3f9061aba141a7d4b2"
format_stamp: "Formatted at 2026-08-07 11:22:04 on dist-test-slave-5l3k"
server_key: "c8d8a907cc27b881c1308b644bf553c3"
server_key_iv: "071556e8b67dd4e2bcd2e7c39d5da18e"
server_key_version: "encryptionkey@0"
I20260807 11:22:04.530144 25294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 11:22:04.534471 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:04.536206 25294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:22:04.536432 25294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "aa8a06dd381f4c3f9061aba141a7d4b2"
format_stamp: "Formatted at 2026-08-07 11:22:04 on dist-test-slave-5l3k"
server_key: "c8d8a907cc27b881c1308b644bf553c3"
server_key_iv: "071556e8b67dd4e2bcd2e7c39d5da18e"
server_key_version: "encryptionkey@0"
I20260807 11:22:04.536674 25294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:04.551409 25294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:04.556735 25294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:04.557106 25294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:04.578660 25294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35691
I20260807 11:22:04.578665 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35691 every 8 connection(s)
I20260807 11:22:04.580416 25294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:04.584892 25364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:04.589529 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25294
I20260807 11:22:04.589758 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:04.590252 2942 external_mini_cluster.cc:1428] Setting key e2f2832de60d92abeb1aa14e61df79e9
I20260807 11:22:04.598140 25364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2: Bootstrap starting.
I20260807 11:22:04.602622 25364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:04.604085 25364 log.cc:824] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:22:04 dist-test-slave-5l3k krb5kdc[25281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:22:04.608319 25364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2: No bootstrap required, opened a new log
I20260807 11:22:04.616915 25364 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260807 11:22:04.617452 25364 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:04.617619 25364 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8a06dd381f4c3f9061aba141a7d4b2, State: Initialized, Role: FOLLOWER
I20260807 11:22:04.618343 25364 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260807 11:22:04.618703 25364 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:04.618875 25364 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:04.619114 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:04.592603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:59019 (local address 127.0.0.1:35691)
0807 11:22:04.593567 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:04.593585 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:04.593645 (+ 60us) server_negotiation.cc:407] Connection header received
0807 11:22:04.596214 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:04.596254 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:04.596787 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:04.597193 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:04.599368 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:04.600826 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:04.602837 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:04.603698 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:04.603873 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:04.609590 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:04.609599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:04.609619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:04.609670 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:04.613485 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:04.614603 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:04.614612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:04.614621 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:04.614831 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:04.615422 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:04.615427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:04.615429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:04.616099 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:04.616486 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:04.617062 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 11:22:04.617511 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":442,"threads_started":1}
I20260807 11:22:04.619102 25364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:04.621829 25364 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260807 11:22:04.622491 25364 leader_election.cc:302] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8a06dd381f4c3f9061aba141a7d4b2; no voters:
I20260807 11:22:04.623379 25364 leader_election.cc:288] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:04.623813 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:04.624039 25369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:04.626197 25364 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:04.630316 25369 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [term 1 LEADER]: Becoming Leader. State: Replica: aa8a06dd381f4c3f9061aba141a7d4b2, State: Running, Role: LEADER
I20260807 11:22:04.631322 25369 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260807 11:22:04.635433 25371 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260807 11:22:04.635679 25372 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8a06dd381f4c3f9061aba141a7d4b2. Latest consensus state: current_term: 1 leader_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a06dd381f4c3f9061aba141a7d4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260807 11:22:04.636011 25372 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:04.636011 25371 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:04.640305 25384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:04.646979 25384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:04.655040 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:04.629425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:42019 (local address 127.0.0.1:35691)
0807 11:22:04.629966 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:04.629974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:04.630000 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:22:04.630282 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:04.630292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:04.630394 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:04.630501 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:04.632438 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:04.633257 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:04.635276 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:04.636563 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:04.636730 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:04.641245 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:04.641255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:04.641261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:04.641311 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:04.648116 (+ 6805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:04.649946 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:04.649956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:04.649962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:04.650207 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:04.651153 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:04.651161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:04.651167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:04.652293 (+ 1126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:04.652636 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:04.654073 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0807 11:22:04.654376 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400}
I20260807 11:22:04.662179 25384 catalog_manager.cc:1347] Generated new cluster ID: ead7be79f60c4d89a165c42fd78196da
I20260807 11:22:04.662343 25384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:04.689464 25384 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:04.691582 25384 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:04.699461 25384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8a06dd381f4c3f9061aba141a7d4b2: Generated new TSK 0
I20260807 11:22:04.700471 25384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:04.708364 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25294
I20260807 11:22:04.709990 25359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:04.809453 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25294
W20260807 11:22:04.829514 25279 connection.cc:535] client connection to 127.0.0.1:35691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35691): Connection reset by peer (error 104)
2026-08-07T11:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:04 dist-test-slave-5l3k krb5kdc[25397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:04 dist-test-slave-5l3k krb5kdc[25397](info): set up 2 sockets
Aug 07 11:22:04 dist-test-slave-5l3k krb5kdc[25397](info): commencing operation
krb5kdc: starting...
W20260807 11:22:06.020601 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.003s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:06 dist-test-slave-5l3k krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:06.430603 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:06.816733 25412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:06.817081 25412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:06.817248 25412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:06.825163 25412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:06.825320 25412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:06.825369 25412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:06.825506 25412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:06.825605 25412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:06.836783 25412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:06.838402 25412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:06.840026 25412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:06.855633 25418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:06.856276 25417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:06.856437 25420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:06.856604 25412 server_base.cc:1034] running on GCE node
I20260807 11:22:06.857597 25412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:06.859474 25412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:06.860766 25412 hybrid_clock.cc:648] HybridClock initialized: now 1786101726860713 us; error 41 us; skew 500 ppm
I20260807 11:22:06.861255 25412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:06 dist-test-slave-5l3k krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:06.864835 25412 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:06.866698 25412 webserver.cc:460] Webserver started at http://127.0.0.1:45169/ using document root <none> and password file <none>
I20260807 11:22:06.867525 25412 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:06.867703 25412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:06.868054 25412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:06.871683 25412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "1d196d139f4c4084934667767773aa61"
format_stamp: "Formatted at 2026-08-07 11:22:06 on dist-test-slave-5l3k"
server_key: "4ec9fa17afb301f804fe2fd2bacbb87c"
server_key_iv: "11f3c4675ee0b0b4318fba52c82a5045"
server_key_version: "encryptionkey@0"
I20260807 11:22:06.872643 25412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "1d196d139f4c4084934667767773aa61"
format_stamp: "Formatted at 2026-08-07 11:22:06 on dist-test-slave-5l3k"
server_key: "4ec9fa17afb301f804fe2fd2bacbb87c"
server_key_iv: "11f3c4675ee0b0b4318fba52c82a5045"
server_key_version: "encryptionkey@0"
I20260807 11:22:06.878930 25412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:22:06.883703 25427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:06.885679 25412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:22:06.885921 25412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "1d196d139f4c4084934667767773aa61"
format_stamp: "Formatted at 2026-08-07 11:22:06 on dist-test-slave-5l3k"
server_key: "4ec9fa17afb301f804fe2fd2bacbb87c"
server_key_iv: "11f3c4675ee0b0b4318fba52c82a5045"
server_key_version: "encryptionkey@0"
I20260807 11:22:06.886186 25412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:06.925325 25412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:06.929879 25412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:06.930217 25412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:06.951690 25412 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260807 11:22:06.951719 25481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260807 11:22:06.953372 25412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:06.958039 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:06.963497 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25412
I20260807 11:22:06.963737 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:06.964301 2942 external_mini_cluster.cc:1428] Setting key 64e3d03d85992bd22ed405f890e19256
I20260807 11:22:06.971525 25482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61: Bootstrap starting.
I20260807 11:22:06.975853 25482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:06.977315 25482 log.cc:824] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:06.980870 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:06.966601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:47773 (local address 127.0.0.1:46593)
0807 11:22:06.967501 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:06.967515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:06.967567 (+ 52us) server_negotiation.cc:407] Connection header received
0807 11:22:06.969630 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:06.969662 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:06.970189 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:06.970710 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:06.971369 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:06.972882 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:06.975141 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:06.975988 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:06.976273 (+ 285us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:06.977799 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:06.977808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:06.977827 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:06.978810 (+ 983us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:06.978930 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:06.978979 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:06.979220 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:06.979562 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 11:22:06.979770 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":394,"threads_started":1}
I20260807 11:22:06.981262 25482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61: No bootstrap required, opened a new log
I20260807 11:22:06.985595 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:06.990195 25482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260807 11:22:06.990886 25482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:06.991083 25482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d196d139f4c4084934667767773aa61, State: Initialized, Role: FOLLOWER
I20260807 11:22:06.991950 25482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260807 11:22:06.992318 25482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:06.992498 25482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:06.992722 25482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:06.996275 25482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260807 11:22:06.997118 25482 leader_election.cc:302] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d196d139f4c4084934667767773aa61; no voters:
I20260807 11:22:06.998021 25482 leader_election.cc:288] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:06.998179 25492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:06.999210 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:06.990667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:37101 (local address 127.0.0.1:46593)
0807 11:22:06.990941 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:06.990951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:06.991621 (+ 670us) server_negotiation.cc:407] Connection header received
0807 11:22:06.991898 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:06.991903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:06.991998 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:06.992092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:06.992865 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:06.993692 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:06.995781 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:06.996276 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:06.996382 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:06.997987 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:06.997992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:06.998003 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:06.998037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:06.998143 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:06.998179 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:06.998327 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:06.998755 (+ 428us) server_negotiation.cc:299] Negotiation successful
0807 11:22:06.998844 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:22:06.999979 25492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [term 1 LEADER]: Becoming Leader. State: Replica: 1d196d139f4c4084934667767773aa61, State: Running, Role: LEADER
I20260807 11:22:07.000669 25492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260807 11:22:07.000864 25482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:07.004635 25493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d196d139f4c4084934667767773aa61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260807 11:22:07.004622 25494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d196d139f4c4084934667767773aa61. Latest consensus state: current_term: 1 leader_uuid: "1d196d139f4c4084934667767773aa61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d196d139f4c4084934667767773aa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260807 11:22:07.005371 25493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:07.005415 25494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:07.006561 25501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:07.013899 25501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:07.025101 25501 catalog_manager.cc:1347] Generated new cluster ID: 91bcadde33254fc8b725f19e964c5098
I20260807 11:22:07.025283 25501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:07.041095 25501 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:07.042624 25501 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:07.055887 25501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d196d139f4c4084934667767773aa61: Generated new TSK 0
I20260807 11:22:07.057111 25501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:07.073751 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25412
W20260807 11:22:07.075217 25476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:47773
W20260807 11:22:07.076239 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25412: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:07.076452 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25412
W20260807 11:22:07.099022 25395 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-08-07T11:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:07 dist-test-slave-5l3k krb5kdc[25515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:07 dist-test-slave-5l3k krb5kdc[25515](info): set up 2 sockets
Aug 07 11:22:07 dist-test-slave-5l3k krb5kdc[25515](info): commencing operation
krb5kdc: starting...
W20260807 11:22:08.281337 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.002s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:08 dist-test-slave-5l3k krb5kdc[25515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:08.682039 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:09.065834 25529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:09.066187 25529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:09.066298 25529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:09.074046 25529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:09.074211 25529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:09.074262 25529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:09.074329 25529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:09.074376 25529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:09.085079 25529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:09.086555 25529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:09.087980 25529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:09.101539 25537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:09.101529 25535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:09.102285 25534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:09.102353 25529 server_base.cc:1034] running on GCE node
I20260807 11:22:09.103361 25529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:09.105144 25529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:09.106446 25529 hybrid_clock.cc:648] HybridClock initialized: now 1786101729106352 us; error 65 us; skew 500 ppm
I20260807 11:22:09.106978 25529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:09 dist-test-slave-5l3k krb5kdc[25515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:09.110960 25529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:09.112763 25529 webserver.cc:460] Webserver started at http://127.0.0.1:39353/ using document root <none> and password file <none>
I20260807 11:22:09.113643 25529 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:09.113826 25529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:09.114188 25529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:09.117823 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "778a79b03c1849d4b64b71fdef60627b"
format_stamp: "Formatted at 2026-08-07 11:22:09 on dist-test-slave-5l3k"
server_key: "f8037f986eb74ad3c09d92f971794018"
server_key_iv: "47cba7edf84965e9129407f0c54d06b3"
server_key_version: "encryptionkey@0"
I20260807 11:22:09.118875 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "778a79b03c1849d4b64b71fdef60627b"
format_stamp: "Formatted at 2026-08-07 11:22:09 on dist-test-slave-5l3k"
server_key: "f8037f986eb74ad3c09d92f971794018"
server_key_iv: "47cba7edf84965e9129407f0c54d06b3"
server_key_version: "encryptionkey@0"
I20260807 11:22:09.124974 25529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:22:09.128953 25544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:09.130636 25529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:22:09.130867 25529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "778a79b03c1849d4b64b71fdef60627b"
format_stamp: "Formatted at 2026-08-07 11:22:09 on dist-test-slave-5l3k"
server_key: "f8037f986eb74ad3c09d92f971794018"
server_key_iv: "47cba7edf84965e9129407f0c54d06b3"
server_key_version: "encryptionkey@0"
I20260807 11:22:09.131115 25529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:09.137697 25529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:09.143184 25529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:09.143559 25529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:09.164565 25529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40165
I20260807 11:22:09.164613 25598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40165 every 8 connection(s)
I20260807 11:22:09.166195 25529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:09.170778 25599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:09.172811 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25529
I20260807 11:22:09.173043 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:09.173648 2942 external_mini_cluster.cc:1428] Setting key d22955b2449d60f9eab7b8d35b536a32
I20260807 11:22:09.183615 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:09.176077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:40261 (local address 127.0.0.1:40165)
0807 11:22:09.177139 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:09.177158 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:09.177217 (+ 59us) server_negotiation.cc:407] Connection header received
0807 11:22:09.179243 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:09.179284 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:09.179789 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:09.180242 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:09.181679 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:09.181687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:09.181703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:09.181721 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:09.181794 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:09.181831 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:09.182025 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:09.182195 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 11:22:09.182456 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":469,"threads_started":1}
I20260807 11:22:09.186861 25599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b: Bootstrap starting.
I20260807 11:22:09.188164 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:09.191123 25599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:09.192850 25599 log.cc:824] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:09.197146 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:09.194180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:35659 (local address 127.0.0.1:40165)
0807 11:22:09.194396 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:09.194404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:09.194930 (+ 526us) server_negotiation.cc:407] Connection header received
0807 11:22:09.195126 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:09.195130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:09.195186 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:09.195280 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:09.196410 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:09.196415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:09.196418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:09.196435 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:09.196478 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:09.196494 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:09.196595 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:09.196749 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 11:22:09.196806 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:22:09.197312 25599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b: No bootstrap required, opened a new log
I20260807 11:22:09.205772 25599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260807 11:22:09.206226 25599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:09.206319 25599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778a79b03c1849d4b64b71fdef60627b, State: Initialized, Role: FOLLOWER
I20260807 11:22:09.207062 25599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260807 11:22:09.207312 25599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:09.207422 25599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:09.207558 25599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:09.209905 25599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260807 11:22:09.210645 25599 leader_election.cc:302] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778a79b03c1849d4b64b71fdef60627b; no voters:
I20260807 11:22:09.211371 25599 leader_election.cc:288] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:09.211551 25609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:09.213346 25599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:09.214861 25609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [term 1 LEADER]: Becoming Leader. State: Replica: 778a79b03c1849d4b64b71fdef60627b, State: Running, Role: LEADER
I20260807 11:22:09.215582 25609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260807 11:22:09.219244 25610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778a79b03c1849d4b64b71fdef60627b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } } }
I20260807 11:22:09.219193 25611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778a79b03c1849d4b64b71fdef60627b. Latest consensus state: current_term: 1 leader_uuid: "778a79b03c1849d4b64b71fdef60627b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a79b03c1849d4b64b71fdef60627b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } } }
I20260807 11:22:09.219746 25610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:09.219746 25611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:09.220721 25619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:09.226912 25619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:09.238054 25619 catalog_manager.cc:1347] Generated new cluster ID: fbfe552d0bea460886003f110fa5949c
I20260807 11:22:09.238220 25619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:09.257892 25619 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:09.258396 25619 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:09.273102 25619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778a79b03c1849d4b64b71fdef60627b: Generated new TSK 0
I20260807 11:22:09.274070 25619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:09.289297 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25529
W20260807 11:22:09.290968 25594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:40261
W20260807 11:22:09.291848 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:09.292038 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25529
2026-08-07T11:22:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:09 dist-test-slave-5l3k krb5kdc[25631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:09 dist-test-slave-5l3k krb5kdc[25631](info): set up 2 sockets
Aug 07 11:22:09 dist-test-slave-5l3k krb5kdc[25631](info): commencing operation
krb5kdc: starting...
W20260807 11:22:10.497186 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:10 dist-test-slave-5l3k krb5kdc[25631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:10.910597 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:11.294566 25644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:11.294972 25644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:11.295104 25644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:11.303155 25644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:11.303303 25644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:11.303360 25644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:11.303431 25644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:11.303478 25644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:11.314380 25644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:11.315920 25644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:11.317469 25644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:11.331480 25652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:11.331470 25649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:11.331737 25650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:11.332873 25644 server_base.cc:1034] running on GCE node
I20260807 11:22:11.333890 25644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:11.335783 25644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:11.337078 25644 hybrid_clock.cc:648] HybridClock initialized: now 1786101731337032 us; error 50 us; skew 500 ppm
I20260807 11:22:11.337517 25644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:11 dist-test-slave-5l3k krb5kdc[25631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:11.341485 25644 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:11.343465 25644 webserver.cc:460] Webserver started at http://127.0.0.1:34737/ using document root <none> and password file <none>
I20260807 11:22:11.344347 25644 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:11.344535 25644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:11.344905 25644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:11.348582 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "c3bb1c9cfe184a52909dd46af8660c27"
format_stamp: "Formatted at 2026-08-07 11:22:11 on dist-test-slave-5l3k"
server_key: "129aaa365a77bd868eed86c88bdfacb6"
server_key_iv: "e31c8f60107f393b778a77a6be819a7b"
server_key_version: "encryptionkey@0"
I20260807 11:22:11.349583 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "c3bb1c9cfe184a52909dd46af8660c27"
format_stamp: "Formatted at 2026-08-07 11:22:11 on dist-test-slave-5l3k"
server_key: "129aaa365a77bd868eed86c88bdfacb6"
server_key_iv: "e31c8f60107f393b778a77a6be819a7b"
server_key_version: "encryptionkey@0"
I20260807 11:22:11.356225 25644 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:22:11.360980 25659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:11.363067 25644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:22:11.363356 25644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "c3bb1c9cfe184a52909dd46af8660c27"
format_stamp: "Formatted at 2026-08-07 11:22:11 on dist-test-slave-5l3k"
server_key: "129aaa365a77bd868eed86c88bdfacb6"
server_key_iv: "e31c8f60107f393b778a77a6be819a7b"
server_key_version: "encryptionkey@0"
I20260807 11:22:11.363633 25644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:11.381563 25644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:11.386513 25644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:11.386914 25644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:11.408267 25644 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40269
I20260807 11:22:11.408291 25713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40269 every 8 connection(s)
I20260807 11:22:11.409992 25644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:11.412624 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25644
I20260807 11:22:11.412847 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:11.413470 2942 external_mini_cluster.cc:1428] Setting key 38b0801c705d97aca4c7ace2a1f5869c
I20260807 11:22:11.415762 25714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:22:11 dist-test-slave-5l3k krb5kdc[25631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:22:11.431774 25714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27: Bootstrap starting.
I20260807 11:22:11.435688 25714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:11.436930 25714 log.cc:824] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:11.440840 25714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27: No bootstrap required, opened a new log
I20260807 11:22:11.441484 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:11.415775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:50595 (local address 127.0.0.1:40269)
0807 11:22:11.416763 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:11.416776 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:11.416827 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:22:11.418747 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:11.418778 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:11.419339 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:11.419764 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:11.421959 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:11.423603 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:11.425569 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:11.426347 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:11.426598 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:11.432216 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:11.432227 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:11.432246 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:11.432301 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:11.436086 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:11.437196 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:11.437205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:11.437213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:11.437409 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:11.438117 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:11.438123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:11.438126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:11.438896 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:11.439157 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:11.439853 (+ 696us) server_negotiation.cc:299] Negotiation successful
0807 11:22:11.440247 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":445,"threads_started":1}
I20260807 11:22:11.446244 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:11.451710 25714 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260807 11:22:11.452346 25714 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:11.452482 25714 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bb1c9cfe184a52909dd46af8660c27, State: Initialized, Role: FOLLOWER
I20260807 11:22:11.453400 25714 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260807 11:22:11.453764 25714 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:11.453956 25714 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:11.454165 25714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:11.457378 25714 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260807 11:22:11.458079 25714 leader_election.cc:302] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3bb1c9cfe184a52909dd46af8660c27; no voters:
I20260807 11:22:11.458966 25714 leader_election.cc:288] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:11.459138 25725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:11.460407 25725 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [term 1 LEADER]: Becoming Leader. State: Replica: c3bb1c9cfe184a52909dd46af8660c27, State: Running, Role: LEADER
I20260807 11:22:11.461190 25725 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260807 11:22:11.462046 25714 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:11.465327 25726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3bb1c9cfe184a52909dd46af8660c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260807 11:22:11.466022 25726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:11.465848 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3bb1c9cfe184a52909dd46af8660c27. Latest consensus state: current_term: 1 leader_uuid: "c3bb1c9cfe184a52909dd46af8660c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb1c9cfe184a52909dd46af8660c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260807 11:22:11.466281 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:11.475703 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:11.452264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:42733 (local address 127.0.0.1:40269)
0807 11:22:11.452546 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:11.452552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:11.453115 (+ 563us) server_negotiation.cc:407] Connection header received
0807 11:22:11.453304 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:11.453309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:11.453415 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:11.453552 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:11.455416 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:11.456645 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:11.458417 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:11.459201 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:11.459414 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:11.463218 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:11.463230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:11.463237 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:11.463309 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:11.468958 (+ 5649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:11.471354 (+ 2396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:11.471364 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:11.471371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:11.471650 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:11.472537 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:11.472545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:11.472550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:11.473181 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:11.474384 (+ 1203us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:11.474715 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:22:11.475038 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 11:22:11.478139 25732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:11.485065 25732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:11.497515 25732 catalog_manager.cc:1347] Generated new cluster ID: dbb9f112e7304d45be223183019c9929
I20260807 11:22:11.497676 25732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:11.509441 25732 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:11.511274 25732 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:11.529421 25732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3bb1c9cfe184a52909dd46af8660c27: Generated new TSK 0
I20260807 11:22:11.530555 25732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:11.550160 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25644
I20260807 11:22:11.551684 25709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:11.649686 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25644
W20260807 11:22:11.672715 25629 connection.cc:535] client connection to 127.0.0.1:40269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40269): Connection reset by peer (error 104)
2026-08-07T11:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:11 dist-test-slave-5l3k krb5kdc[25750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:11 dist-test-slave-5l3k krb5kdc[25750](info): set up 2 sockets
Aug 07 11:22:11 dist-test-slave-5l3k krb5kdc[25750](info): commencing operation
krb5kdc: starting...
W20260807 11:22:12.870076 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.002s sys 0.069s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:13 dist-test-slave-5l3k krb5kdc[25750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:13.282160 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:13.681926 25763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:13.682333 25763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:13.682606 25763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:13.690899 25763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:13.691064 25763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:13.691145 25763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:13.691270 25763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:13.691349 25763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:13.702991 25763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:13.704644 25763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:13.706315 25763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:13.721277 25769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:13.721537 25771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:13.721879 25768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:13.722363 25763 server_base.cc:1034] running on GCE node
I20260807 11:22:13.723459 25763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:13.725257 25763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:13.726672 25763 hybrid_clock.cc:648] HybridClock initialized: now 1786101733726617 us; error 71 us; skew 500 ppm
I20260807 11:22:13.727196 25763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:13 dist-test-slave-5l3k krb5kdc[25750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:13.731271 25763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:13.733310 25763 webserver.cc:460] Webserver started at http://127.0.0.1:36689/ using document root <none> and password file <none>
I20260807 11:22:13.734232 25763 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:13.734436 25763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:13.734867 25763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:13.738917 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "24ef5e7ce0404b3e9e40d963e5848635"
format_stamp: "Formatted at 2026-08-07 11:22:13 on dist-test-slave-5l3k"
server_key: "dcbf824a440520dc345f16f8ce4a3909"
server_key_iv: "46aecbc901d2c50b5d6e7eb3ff05711e"
server_key_version: "encryptionkey@0"
I20260807 11:22:13.740023 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "24ef5e7ce0404b3e9e40d963e5848635"
format_stamp: "Formatted at 2026-08-07 11:22:13 on dist-test-slave-5l3k"
server_key: "dcbf824a440520dc345f16f8ce4a3909"
server_key_iv: "46aecbc901d2c50b5d6e7eb3ff05711e"
server_key_version: "encryptionkey@0"
I20260807 11:22:13.747087 25763 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 11:22:13.751827 25779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:13.753739 25763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:22:13.754031 25763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "24ef5e7ce0404b3e9e40d963e5848635"
format_stamp: "Formatted at 2026-08-07 11:22:13 on dist-test-slave-5l3k"
server_key: "dcbf824a440520dc345f16f8ce4a3909"
server_key_iv: "46aecbc901d2c50b5d6e7eb3ff05711e"
server_key_version: "encryptionkey@0"
I20260807 11:22:13.754312 25763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:13.770287 25763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:13.776015 25763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:13.776386 25763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:13.799654 25763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40489
I20260807 11:22:13.799675 25834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40489 every 8 connection(s)
I20260807 11:22:13.801599 25763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:13.804759 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25763
I20260807 11:22:13.805056 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:13.805600 2942 external_mini_cluster.cc:1428] Setting key f695a8606e2f0af61e753cd2e4601323
I20260807 11:22:13.806948 25835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:13.825029 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:13.808189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:52385 (local address 127.0.0.1:40489)
0807 11:22:13.809252 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:13.809272 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:13.809339 (+ 67us) server_negotiation.cc:407] Connection header received
0807 11:22:13.811493 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:13.811536 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:13.812328 (+ 792us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:13.812932 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:13.813608 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:13.816036 (+ 2428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:13.817964 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:13.819222 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:13.819474 (+ 252us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:13.821180 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:13.821194 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:13.821221 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:13.822181 (+ 960us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:13.822323 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:13.822373 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:13.822627 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:13.823060 (+ 433us) server_negotiation.cc:299] Negotiation successful
0807 11:22:13.823309 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":482,"threads_started":1}
I20260807 11:22:13.825176 25835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635: Bootstrap starting.
I20260807 11:22:13.829353 25835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:13.831027 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:13.831313 25835 log.cc:824] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:13.835988 25835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635: No bootstrap required, opened a new log
I20260807 11:22:13.845819 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:13.837004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:56631 (local address 127.0.0.1:40489)
0807 11:22:13.837230 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:13.837236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:13.837783 (+ 547us) server_negotiation.cc:407] Connection header received
0807 11:22:13.837991 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:13.837997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:13.838095 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:13.838241 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:13.838996 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:13.840323 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:13.842209 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:13.843037 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:13.843225 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:13.844742 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:13.844750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:13.844755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:13.844785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:13.844848 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:13.844881 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:13.845022 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:13.845323 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 11:22:13.845398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:22:13.847465 25835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260807 11:22:13.848171 25835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:13.848337 25835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ef5e7ce0404b3e9e40d963e5848635, State: Initialized, Role: FOLLOWER
I20260807 11:22:13.849076 25835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260807 11:22:13.849411 25835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:13.849596 25835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:13.849812 25835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:13.852528 25835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260807 11:22:13.853247 25835 leader_election.cc:302] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ef5e7ce0404b3e9e40d963e5848635; no voters:
I20260807 11:22:13.854049 25835 leader_election.cc:288] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:13.854326 25846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:13.856317 25846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [term 1 LEADER]: Becoming Leader. State: Replica: 24ef5e7ce0404b3e9e40d963e5848635, State: Running, Role: LEADER
I20260807 11:22:13.856967 25835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:13.857193 25846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260807 11:22:13.860879 25848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ef5e7ce0404b3e9e40d963e5848635. Latest consensus state: current_term: 1 leader_uuid: "24ef5e7ce0404b3e9e40d963e5848635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260807 11:22:13.861438 25848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:13.862216 25847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ef5e7ce0404b3e9e40d963e5848635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef5e7ce0404b3e9e40d963e5848635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260807 11:22:13.862713 25847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:13.868947 25855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:13.874800 25855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:13.888690 25855 catalog_manager.cc:1347] Generated new cluster ID: 9b12f0d704c54ef59975ee48f2c24fd7
I20260807 11:22:13.888872 25855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:13.902040 25855 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:13.903707 25855 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:13.922050 25855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ef5e7ce0404b3e9e40d963e5848635: Generated new TSK 0
I20260807 11:22:13.923182 25855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:13.933104 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25763
W20260807 11:22:13.934680 25829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:52385
W20260807 11:22:13.935672 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25763: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:13.935861 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25763
W20260807 11:22:13.959336 25747 connection.cc:535] client connection to 127.0.0.1:40489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40489): Connection reset by peer (error 104)
2026-08-07T11:22:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:14 dist-test-slave-5l3k krb5kdc[25871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:14 dist-test-slave-5l3k krb5kdc[25871](info): set up 2 sockets
Aug 07 11:22:14 dist-test-slave-5l3k krb5kdc[25871](info): commencing operation
krb5kdc: starting...
W20260807 11:22:15.149735 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:15 dist-test-slave-5l3k krb5kdc[25871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:22:15.562621 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:15.957276 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:15.957640 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:15.957846 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:15.966008 25885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:15.966179 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:15.966269 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:15.966389 25885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:15.966445 25885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:15.977965 25885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:15.979734 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:15.981359 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:15.995827 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:15.995762 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:15.995785 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:15.997328 25885 server_base.cc:1034] running on GCE node
I20260807 11:22:15.998293 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:16.000125 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:16.001405 25885 hybrid_clock.cc:648] HybridClock initialized: now 1786101736001340 us; error 53 us; skew 500 ppm
I20260807 11:22:16.001818 25885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:16 dist-test-slave-5l3k krb5kdc[25871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:16.005656 25885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:22:16.007627 25885 webserver.cc:460] Webserver started at http://127.0.0.1:41875/ using document root <none> and password file <none>
I20260807 11:22:16.008495 25885 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:16.008656 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:16.009001 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:16.012792 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "10355f6009ee44e9959dc526f80f73e6"
format_stamp: "Formatted at 2026-08-07 11:22:16 on dist-test-slave-5l3k"
server_key: "bae0311b606c77a8f261fa0cac1d6b88"
server_key_iv: "d9d754168d882d74d80de81860a2b379"
server_key_version: "encryptionkey@0"
I20260807 11:22:16.013890 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "10355f6009ee44e9959dc526f80f73e6"
format_stamp: "Formatted at 2026-08-07 11:22:16 on dist-test-slave-5l3k"
server_key: "bae0311b606c77a8f261fa0cac1d6b88"
server_key_iv: "d9d754168d882d74d80de81860a2b379"
server_key_version: "encryptionkey@0"
I20260807 11:22:16.020732 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260807 11:22:16.025473 25900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:16.027498 25885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 11:22:16.027725 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "10355f6009ee44e9959dc526f80f73e6"
format_stamp: "Formatted at 2026-08-07 11:22:16 on dist-test-slave-5l3k"
server_key: "bae0311b606c77a8f261fa0cac1d6b88"
server_key_iv: "d9d754168d882d74d80de81860a2b379"
server_key_version: "encryptionkey@0"
I20260807 11:22:16.027948 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:16.035375 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:16.040594 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:16.040925 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:16.063668 25885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46115
I20260807 11:22:16.063704 25955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46115 every 8 connection(s)
I20260807 11:22:16.065472 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:16.070353 25956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:16.075340 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 25885
I20260807 11:22:16.075572 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:16.076180 2942 external_mini_cluster.cc:1428] Setting key 90ca1b314a465d82d84bd026863741a2
I20260807 11:22:16.085726 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:16.078593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:60331 (local address 127.0.0.1:46115)
0807 11:22:16.079577 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:16.079592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:16.079653 (+ 61us) server_negotiation.cc:407] Connection header received
0807 11:22:16.081509 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:16.081540 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:16.082044 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:16.082563 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:16.083798 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:16.083805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:16.083822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:16.083839 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:16.083911 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:16.083933 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:16.084136 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:16.084257 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 11:22:16.084528 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":443,"threads_started":1}
I20260807 11:22:16.086395 25956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6: Bootstrap starting.
I20260807 11:22:16.090299 25956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:16.090698 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:16.091974 25956 log.cc:824] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:16.096769 25956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6: No bootstrap required, opened a new log
I20260807 11:22:16.100016 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:16.096866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:40309 (local address 127.0.0.1:46115)
0807 11:22:16.097124 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:16.097131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:16.097474 (+ 343us) server_negotiation.cc:407] Connection header received
0807 11:22:16.097704 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:16.097709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:16.097778 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:16.097881 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:16.099080 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:16.099085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:16.099088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:16.099106 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:16.099154 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:16.099171 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:16.099274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:16.099498 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 11:22:16.099629 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 11:22:16.106431 25956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260807 11:22:16.107049 25956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:16.107187 25956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10355f6009ee44e9959dc526f80f73e6, State: Initialized, Role: FOLLOWER
I20260807 11:22:16.107880 25956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260807 11:22:16.108183 25956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:16.108348 25956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:16.108546 25956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:16.111158 25956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260807 11:22:16.111876 25956 leader_election.cc:302] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10355f6009ee44e9959dc526f80f73e6; no voters:
I20260807 11:22:16.112622 25956 leader_election.cc:288] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:16.112775 25966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:16.113543 25966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [term 1 LEADER]: Becoming Leader. State: Replica: 10355f6009ee44e9959dc526f80f73e6, State: Running, Role: LEADER
I20260807 11:22:16.114243 25966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260807 11:22:16.115180 25956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:16.118005 25967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10355f6009ee44e9959dc526f80f73e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } } }
I20260807 11:22:16.117884 25968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10355f6009ee44e9959dc526f80f73e6. Latest consensus state: current_term: 1 leader_uuid: "10355f6009ee44e9959dc526f80f73e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10355f6009ee44e9959dc526f80f73e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } } }
I20260807 11:22:16.118647 25967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:16.118672 25968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:16.120218 25975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:16.128158 25975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:16.139653 25975 catalog_manager.cc:1347] Generated new cluster ID: fe1aad43fb7047d3a85144e8e83e12d6
I20260807 11:22:16.139837 25975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:16.163195 25975 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:16.163753 25975 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:16.172958 25975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10355f6009ee44e9959dc526f80f73e6: Generated new TSK 0
I20260807 11:22:16.174026 25975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:16.183344 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25885
W20260807 11:22:16.184708 25951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:60331
W20260807 11:22:16.185616 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 25885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:16.185801 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 25885
2026-08-07T11:22:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:16 dist-test-slave-5l3k krb5kdc[25987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:16 dist-test-slave-5l3k krb5kdc[25987](info): set up 2 sockets
Aug 07 11:22:16 dist-test-slave-5l3k krb5kdc[25987](info): commencing operation
krb5kdc: starting...
W20260807 11:22:17.425675 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:17 dist-test-slave-5l3k krb5kdc[25987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:17.843052 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42521
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:42521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:18.229444 26000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:18.229902 26000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:18.230093 26000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:18.238065 26000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:18.238288 26000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:18.238422 26000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:18.238565 26000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:18.238649 26000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:18.249648 26000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:42521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:18.251308 26000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:18.252888 26000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:18.266757 26005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:18.266986 26007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:18.267441 26009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:18.268512 26000 server_base.cc:1034] running on GCE node
I20260807 11:22:18.269449 26000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:18.271276 26000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:18.272535 26000 hybrid_clock.cc:648] HybridClock initialized: now 1786101738272477 us; error 51 us; skew 500 ppm
I20260807 11:22:18.273000 26000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:18 dist-test-slave-5l3k krb5kdc[25987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:18.282130 26000 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:18.284343 26000 webserver.cc:460] Webserver started at http://127.2.223.190:34965/ using document root <none> and password file <none>
I20260807 11:22:18.285290 26000 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:18.285494 26000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:18.285897 26000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:18.289974 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "546ffe27e190431ab00947844a4c0a24"
format_stamp: "Formatted at 2026-08-07 11:22:18 on dist-test-slave-5l3k"
server_key: "47f1173ff061ea2d495fd3b683758ebb"
server_key_iv: "e6398f9b477ea071773214013fe13aac"
server_key_version: "encryptionkey@0"
I20260807 11:22:18.291188 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "546ffe27e190431ab00947844a4c0a24"
format_stamp: "Formatted at 2026-08-07 11:22:18 on dist-test-slave-5l3k"
server_key: "47f1173ff061ea2d495fd3b683758ebb"
server_key_iv: "e6398f9b477ea071773214013fe13aac"
server_key_version: "encryptionkey@0"
I20260807 11:22:18.298542 26000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260807 11:22:18.303589 26018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:18.305509 26000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260807 11:22:18.305824 26000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "546ffe27e190431ab00947844a4c0a24"
format_stamp: "Formatted at 2026-08-07 11:22:18 on dist-test-slave-5l3k"
server_key: "47f1173ff061ea2d495fd3b683758ebb"
server_key_iv: "e6398f9b477ea071773214013fe13aac"
server_key_version: "encryptionkey@0"
I20260807 11:22:18.306181 26000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:18.332073 26000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:18.337524 26000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:18.337922 26000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:18.360249 26000 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:42521
I20260807 11:22:18.360280 26072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:42521 every 8 connection(s)
I20260807 11:22:18.361949 26000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:18.365897 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26000
I20260807 11:22:18.366123 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:18.366780 2942 external_mini_cluster.cc:1428] Setting key 6ddb3d15da4bc0076375f99ca95fa491
I20260807 11:22:18.367214 26073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:18.384692 26073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24: Bootstrap starting.
Aug 07 11:22:18 dist-test-slave-5l3k krb5kdc[25987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:18.389299 26073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:18.391111 26073 log.cc:824] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:18.395280 26073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24: No bootstrap required, opened a new log
I20260807 11:22:18.398837 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:18.369186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49697 (local address 127.2.223.190:42521)
0807 11:22:18.370264 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:18.370278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:18.370344 (+ 66us) server_negotiation.cc:407] Connection header received
0807 11:22:18.372928 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:18.372964 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:18.373588 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:18.374068 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:18.376357 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:18.378637 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:18.380467 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:18.381888 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:18.382194 (+ 306us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:18.387891 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:18.387904 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:18.387931 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:18.388007 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:18.392599 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:18.393771 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:18.393781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:18.393791 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:18.394045 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:18.394892 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:18.394897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:18.394900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:18.395724 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:18.396135 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:18.396649 (+ 514us) server_negotiation.cc:299] Negotiation successful
0807 11:22:18.397075 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":491,"threads_started":1}
I20260807 11:22:18.404479 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:18.405817 26073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } }
I20260807 11:22:18.406319 26073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:18.406456 26073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546ffe27e190431ab00947844a4c0a24, State: Initialized, Role: FOLLOWER
I20260807 11:22:18.407158 26073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } }
I20260807 11:22:18.407440 26073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:18.407631 26073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:18.407864 26073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:18.411275 26073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } }
I20260807 11:22:18.411913 26073 leader_election.cc:302] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 546ffe27e190431ab00947844a4c0a24; no voters:
I20260807 11:22:18.412792 26073 leader_election.cc:288] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:18.412921 26085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:18.414816 26085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [term 1 LEADER]: Becoming Leader. State: Replica: 546ffe27e190431ab00947844a4c0a24, State: Running, Role: LEADER
I20260807 11:22:18.415639 26085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } }
I20260807 11:22:18.415957 26073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:18.422155 26087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 546ffe27e190431ab00947844a4c0a24. Latest consensus state: current_term: 1 leader_uuid: "546ffe27e190431ab00947844a4c0a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } } }
I20260807 11:22:18.422886 26087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:18.423650 26086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "546ffe27e190431ab00947844a4c0a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546ffe27e190431ab00947844a4c0a24" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 42521 } } }
I20260807 11:22:18.424069 26086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:18.429597 26092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:18.431169 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:18.409677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50059 (local address 127.2.223.190:42521)
0807 11:22:18.409901 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:18.409909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:18.410280 (+ 371us) server_negotiation.cc:407] Connection header received
0807 11:22:18.410552 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:18.410560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:18.410649 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:18.410766 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:18.412656 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:18.413985 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:18.415807 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:18.416419 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:18.416597 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:18.420408 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:18.420419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:18.420428 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:18.420489 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:18.425722 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:18.427541 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:18.427552 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:18.427558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:18.427846 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:18.428790 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:18.428798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:18.428804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:18.429437 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:18.429897 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:18.430407 (+ 510us) server_negotiation.cc:299] Negotiation successful
0807 11:22:18.430690 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:22:18.436923 26092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:18.448709 26092 catalog_manager.cc:1347] Generated new cluster ID: 2218f31f3c24490c8195f374c36c12b2
I20260807 11:22:18.448891 26092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:18.473440 26092 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:18.475095 26092 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:18.489379 26092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 546ffe27e190431ab00947844a4c0a24: Generated new TSK 0
I20260807 11:22:18.490509 26092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:18.500813 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26000
I20260807 11:22:18.502815 26068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:18.602669 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26000
W20260807 11:22:18.626071 25985 connection.cc:535] client connection to 127.2.223.190:42521 recv error: Network error: failed to read from TLS socket (remote: 127.2.223.190:42521): Connection reset by peer (error 104)
2026-08-07T11:22:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:18 dist-test-slave-5l3k krb5kdc[26110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:18 dist-test-slave-5l3k krb5kdc[26110](info): set up 2 sockets
Aug 07 11:22:18 dist-test-slave-5l3k krb5kdc[26110](info): commencing operation
krb5kdc: starting...
W20260807 11:22:19.821746 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.007s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:20 dist-test-slave-5l3k krb5kdc[26110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:20.234656 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:34539
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:34539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:20.623323 26123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:20.623775 26123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:20.623950 26123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:20.631748 26123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:20.631971 26123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:20.632112 26123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:20.632225 26123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:20.632316 26123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:20.643167 26123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:34539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:20.644820 26123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:20.646414 26123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:20.660391 26131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:20.660444 26128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:20.660480 26129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:20.661964 26123 server_base.cc:1034] running on GCE node
I20260807 11:22:20.663156 26123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:20.664943 26123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:20.666292 26123 hybrid_clock.cc:648] HybridClock initialized: now 1786101740666221 us; error 49 us; skew 500 ppm
I20260807 11:22:20.666826 26123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:20 dist-test-slave-5l3k krb5kdc[26110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:20.670706 26123 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:20.672636 26123 webserver.cc:460] Webserver started at http://127.2.223.190:40613/ using document root <none> and password file <none>
I20260807 11:22:20.673501 26123 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:20.673691 26123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:20.674067 26123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:20.677774 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "8bdb362b930b4adab2b1456ab5ae198a"
format_stamp: "Formatted at 2026-08-07 11:22:20 on dist-test-slave-5l3k"
server_key: "eba0aed1c67f855aa7a37258155c241a"
server_key_iv: "b1ea630931b1ca72d6cfe3d60e0ce16b"
server_key_version: "encryptionkey@0"
I20260807 11:22:20.678836 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "8bdb362b930b4adab2b1456ab5ae198a"
format_stamp: "Formatted at 2026-08-07 11:22:20 on dist-test-slave-5l3k"
server_key: "eba0aed1c67f855aa7a37258155c241a"
server_key_iv: "b1ea630931b1ca72d6cfe3d60e0ce16b"
server_key_version: "encryptionkey@0"
I20260807 11:22:20.685322 26123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 11:22:20.690065 26138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:20.691968 26123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:22:20.692286 26123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "8bdb362b930b4adab2b1456ab5ae198a"
format_stamp: "Formatted at 2026-08-07 11:22:20 on dist-test-slave-5l3k"
server_key: "eba0aed1c67f855aa7a37258155c241a"
server_key_iv: "b1ea630931b1ca72d6cfe3d60e0ce16b"
server_key_version: "encryptionkey@0"
I20260807 11:22:20.692562 26123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:20.708478 26123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:20.713922 26123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:20.714325 26123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:20.736833 26123 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:34539
I20260807 11:22:20.736863 26195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:34539 every 8 connection(s)
I20260807 11:22:20.738605 26123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:20.743552 26196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:20.746945 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26123
I20260807 11:22:20.747169 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:20.747654 2942 external_mini_cluster.cc:1428] Setting key c18a84fbec55af708d8958723f760e30
I20260807 11:22:20.758085 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a: Bootstrap starting.
I20260807 11:22:20.762797 26196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:20.764189 26196 log.cc:824] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:20.764442 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:20.750037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53889 (local address 127.2.223.190:34539)
0807 11:22:20.751072 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:20.751087 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:20.751137 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:22:20.753059 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:20.753095 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:20.753624 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:20.754064 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:20.754940 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:20.756553 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:20.758786 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:20.759596 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:20.759767 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:20.761431 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:20.761440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:20.761459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:20.762124 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:20.762222 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:20.762256 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:20.762409 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:20.762834 (+ 425us) server_negotiation.cc:299] Negotiation successful
0807 11:22:20.763209 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":504,"threads_started":1}
I20260807 11:22:20.768122 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a: No bootstrap required, opened a new log
I20260807 11:22:20.769441 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:20.778322 26196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } }
I20260807 11:22:20.779009 26196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:20.779141 26196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdb362b930b4adab2b1456ab5ae198a, State: Initialized, Role: FOLLOWER
I20260807 11:22:20.779944 26196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } }
I20260807 11:22:20.780274 26196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:20.780454 26196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:20.780681 26196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:20.782763 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:20.774561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47335 (local address 127.2.223.190:34539)
0807 11:22:20.774844 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:20.774855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:20.775419 (+ 564us) server_negotiation.cc:407] Connection header received
0807 11:22:20.775648 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:20.775652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:20.775729 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:20.775819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:20.776603 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:20.777440 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:20.779614 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:20.780211 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:20.780393 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:20.781866 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:20.781872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:20.781876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:20.781903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:20.781952 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:20.781971 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:20.782136 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:20.782351 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 11:22:20.782423 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 11:22:20.784008 26196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } }
I20260807 11:22:20.784796 26196 leader_election.cc:302] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bdb362b930b4adab2b1456ab5ae198a; no voters:
I20260807 11:22:20.785564 26196 leader_election.cc:288] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:20.785748 26208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:20.786913 26208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [term 1 LEADER]: Becoming Leader. State: Replica: 8bdb362b930b4adab2b1456ab5ae198a, State: Running, Role: LEADER
I20260807 11:22:20.787585 26208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } }
I20260807 11:22:20.788518 26196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:20.790970 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bdb362b930b4adab2b1456ab5ae198a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } } }
I20260807 11:22:20.791523 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:20.792459 26213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:20.792290 26210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bdb362b930b4adab2b1456ab5ae198a. Latest consensus state: current_term: 1 leader_uuid: "8bdb362b930b4adab2b1456ab5ae198a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdb362b930b4adab2b1456ab5ae198a" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 34539 } } }
I20260807 11:22:20.792898 26210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:20.800509 26213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:20.813019 26213 catalog_manager.cc:1347] Generated new cluster ID: 51506470303743e3be9bfdf8eaa9afb3
I20260807 11:22:20.813176 26213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:20.824586 26213 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:20.826107 26213 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:20.835445 26213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bdb362b930b4adab2b1456ab5ae198a: Generated new TSK 0
I20260807 11:22:20.836520 26213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:20.856580 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26123
W20260807 11:22:20.858071 26190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53889
W20260807 11:22:20.859305 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:20.859519 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26123
W20260807 11:22:20.882535 26108 connection.cc:535] client connection to 127.2.223.190:34539 recv error: Network error: failed to read from TLS socket (remote: 127.2.223.190:34539): Connection reset by peer (error 104)
2026-08-07T11:22:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:21 dist-test-slave-5l3k krb5kdc[26231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:21 dist-test-slave-5l3k krb5kdc[26231](info): set up 2 sockets
Aug 07 11:22:21 dist-test-slave-5l3k krb5kdc[26231](info): commencing operation
krb5kdc: starting...
W20260807 11:22:22.093912 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.001s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:22 dist-test-slave-5l3k krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:22.506232 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46311
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:22.903301 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:22.903683 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:22.903875 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:22.911827 26245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:22.911974 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:22.912185 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:22.912336 26245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:22.912425 26245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:22.923590 26245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:22.925233 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:22.926928 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:22.941349 26253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:22.941323 26250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:22.941277 26251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:22.942255 26245 server_base.cc:1034] running on GCE node
I20260807 11:22:22.943336 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:22.945153 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:22.946440 26245 hybrid_clock.cc:648] HybridClock initialized: now 1786101742946388 us; error 47 us; skew 500 ppm
I20260807 11:22:22.946970 26245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:22 dist-test-slave-5l3k krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:22.950722 26245 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:22.952610 26245 webserver.cc:460] Webserver started at http://127.2.223.190:37529/ using document root <none> and password file <none>
I20260807 11:22:22.953465 26245 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:22.953647 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:22.954006 26245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:22.957806 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "579eb935df3949de95edd79a616415be"
format_stamp: "Formatted at 2026-08-07 11:22:22 on dist-test-slave-5l3k"
server_key: "59d16d137ac132343e577982d312507c"
server_key_iv: "4febd473ba7a4a9179ed3c1e8f3d6ea8"
server_key_version: "encryptionkey@0"
I20260807 11:22:22.959198 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "579eb935df3949de95edd79a616415be"
format_stamp: "Formatted at 2026-08-07 11:22:22 on dist-test-slave-5l3k"
server_key: "59d16d137ac132343e577982d312507c"
server_key_iv: "4febd473ba7a4a9179ed3c1e8f3d6ea8"
server_key_version: "encryptionkey@0"
I20260807 11:22:22.966766 26245 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260807 11:22:22.971206 26260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:22.973014 26245 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:22:22.973307 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "579eb935df3949de95edd79a616415be"
format_stamp: "Formatted at 2026-08-07 11:22:22 on dist-test-slave-5l3k"
server_key: "59d16d137ac132343e577982d312507c"
server_key_iv: "4febd473ba7a4a9179ed3c1e8f3d6ea8"
server_key_version: "encryptionkey@0"
I20260807 11:22:22.973577 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:22.980790 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:22.985836 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:22.986227 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:23.007774 26245 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:46311
I20260807 11:22:23.007814 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:46311 every 8 connection(s)
I20260807 11:22:23.009420 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:23.013972 26315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:23.017915 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26245
I20260807 11:22:23.018143 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:23.018688 2942 external_mini_cluster.cc:1428] Setting key 73fb473950eb181e147d53a8f9387a56
I20260807 11:22:23.028481 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:23.021033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44753 (local address 127.2.223.190:46311)
0807 11:22:23.022025 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:23.022040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:23.022093 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:22:23.024084 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:23.024115 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:23.024591 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:23.025169 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:23.026470 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:23.026532 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:23.026556 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:23.026590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:23.026662 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:23.026686 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:23.026961 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:23.027142 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 11:22:23.027368 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":464,"threads_started":1}
I20260807 11:22:23.028609 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be: Bootstrap starting.
I20260807 11:22:23.032848 26315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:23.033300 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:23.034188 26315 log.cc:824] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:23.038265 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be: No bootstrap required, opened a new log
I20260807 11:22:23.042006 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:23.038795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56989 (local address 127.2.223.190:46311)
0807 11:22:23.039079 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:23.039086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:23.039557 (+ 471us) server_negotiation.cc:407] Connection header received
0807 11:22:23.039785 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:23.039789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:23.039847 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:23.039941 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:23.041201 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:23.041205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:23.041209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:23.041228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:23.041292 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:23.041310 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:23.041422 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:23.041588 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 11:22:23.041649 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:22:23.048089 26315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } }
I20260807 11:22:23.048625 26315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:23.048758 26315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579eb935df3949de95edd79a616415be, State: Initialized, Role: FOLLOWER
I20260807 11:22:23.049448 26315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } }
I20260807 11:22:23.049758 26315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:23.049929 26315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:23.050138 26315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:23.052736 26315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } }
I20260807 11:22:23.053435 26315 leader_election.cc:302] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579eb935df3949de95edd79a616415be; no voters:
I20260807 11:22:23.054128 26315 leader_election.cc:288] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:23.054343 26326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:23.056200 26326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [term 1 LEADER]: Becoming Leader. State: Replica: 579eb935df3949de95edd79a616415be, State: Running, Role: LEADER
I20260807 11:22:23.056900 26326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } }
I20260807 11:22:23.058028 26315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:23.060487 26328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579eb935df3949de95edd79a616415be. Latest consensus state: current_term: 1 leader_uuid: "579eb935df3949de95edd79a616415be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } } }
I20260807 11:22:23.060968 26328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:23.060586 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579eb935df3949de95edd79a616415be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579eb935df3949de95edd79a616415be" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46311 } } }
I20260807 11:22:23.061172 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:23.063691 26331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:23.070739 26331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:23.081826 26331 catalog_manager.cc:1347] Generated new cluster ID: 33d43579872244d0a10f9d24a56633e6
I20260807 11:22:23.081979 26331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:23.097059 26331 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:23.097657 26331 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:23.107261 26331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 579eb935df3949de95edd79a616415be: Generated new TSK 0
I20260807 11:22:23.108630 26331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:23.122035 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26245
W20260807 11:22:23.123377 26308 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44753
W20260807 11:22:23.124239 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:23.124424 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26245
2026-08-07T11:22:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:23 dist-test-slave-5l3k krb5kdc[26349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:23 dist-test-slave-5l3k krb5kdc[26349](info): set up 2 sockets
Aug 07 11:22:23 dist-test-slave-5l3k krb5kdc[26349](info): commencing operation
krb5kdc: starting...
W20260807 11:22:24.349390 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.001s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:24 dist-test-slave-5l3k krb5kdc[26349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:24.758404 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39781
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:25.148041 26364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:25.148401 26364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:25.148502 26364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:25.156306 26364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:25.156469 26364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:25.156520 26364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:25.156577 26364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:25.156625 26364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:25.167740 26364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:25.169224 26364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:25.170692 26364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:25.184607 26371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:25.184753 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:25.184648 26370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:25.185912 26364 server_base.cc:1034] running on GCE node
I20260807 11:22:25.186941 26364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:25.188639 26364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:25.189921 26364 hybrid_clock.cc:648] HybridClock initialized: now 1786101745189832 us; error 69 us; skew 500 ppm
I20260807 11:22:25.190541 26364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:25 dist-test-slave-5l3k krb5kdc[26349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:25.197999 26364 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:25.200080 26364 webserver.cc:460] Webserver started at http://127.2.223.190:33523/ using document root <none> and password file <none>
I20260807 11:22:25.200913 26364 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:25.201067 26364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:25.201444 26364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:25.204968 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "a5221458556944d18125cfd85f217737"
format_stamp: "Formatted at 2026-08-07 11:22:25 on dist-test-slave-5l3k"
server_key: "4955f17ecb06d388b61bf2497b23c0e4"
server_key_iv: "ec2c8c0d2f70ff930a238efdea65dc29"
server_key_version: "encryptionkey@0"
I20260807 11:22:25.205935 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "a5221458556944d18125cfd85f217737"
format_stamp: "Formatted at 2026-08-07 11:22:25 on dist-test-slave-5l3k"
server_key: "4955f17ecb06d388b61bf2497b23c0e4"
server_key_iv: "ec2c8c0d2f70ff930a238efdea65dc29"
server_key_version: "encryptionkey@0"
I20260807 11:22:25.212280 26364 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:22:25.216661 26380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:25.218358 26364 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260807 11:22:25.218672 26364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "a5221458556944d18125cfd85f217737"
format_stamp: "Formatted at 2026-08-07 11:22:25 on dist-test-slave-5l3k"
server_key: "4955f17ecb06d388b61bf2497b23c0e4"
server_key_iv: "ec2c8c0d2f70ff930a238efdea65dc29"
server_key_version: "encryptionkey@0"
I20260807 11:22:25.218912 26364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:25.233630 26364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:25.238369 26364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:25.238752 26364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:25.259977 26364 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:39781
I20260807 11:22:25.259984 26434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:39781 every 8 connection(s)
I20260807 11:22:25.261673 26364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:25.266254 26435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:25.271070 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26364
I20260807 11:22:25.271301 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:25.271782 2942 external_mini_cluster.cc:1428] Setting key 637fdb54e12cf9a29c31d8635109eace
I20260807 11:22:25.280941 26435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737: Bootstrap starting.
I20260807 11:22:25.284755 26435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:25.286002 26435 log.cc:824] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:22:25 dist-test-slave-5l3k krb5kdc[26349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:25.289577 26435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737: No bootstrap required, opened a new log
I20260807 11:22:25.297858 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:25.273915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44409 (local address 127.2.223.190:39781)
0807 11:22:25.274941 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:25.274956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:25.275007 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:22:25.276867 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:25.276902 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:25.277476 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:25.278037 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:25.279822 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:25.281344 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:25.282929 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:25.283718 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:25.283964 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:25.288690 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:25.288700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:25.288719 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:25.288767 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:25.292487 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:25.293467 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:25.293476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:25.293484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:25.293697 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:25.294202 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:25.294207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:25.294209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:25.294993 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:25.295394 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:25.295966 (+ 572us) server_negotiation.cc:299] Negotiation successful
0807 11:22:25.296528 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":501,"threads_started":1}
I20260807 11:22:25.298796 26435 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } }
I20260807 11:22:25.299347 26435 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:25.299477 26435 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5221458556944d18125cfd85f217737, State: Initialized, Role: FOLLOWER
I20260807 11:22:25.300230 26435 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } }
I20260807 11:22:25.300491 26435 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:25.300705 26435 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:25.300915 26435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:25.303118 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:25.304244 26435 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } }
I20260807 11:22:25.304991 26435 leader_election.cc:302] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5221458556944d18125cfd85f217737; no voters:
I20260807 11:22:25.305791 26435 leader_election.cc:288] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:25.305946 26440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:25.308987 26435 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:25.310875 26440 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [term 1 LEADER]: Becoming Leader. State: Replica: a5221458556944d18125cfd85f217737, State: Running, Role: LEADER
I20260807 11:22:25.311663 26440 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } }
I20260807 11:22:25.315196 26442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5221458556944d18125cfd85f217737. Latest consensus state: current_term: 1 leader_uuid: "a5221458556944d18125cfd85f217737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } } }
I20260807 11:22:25.315685 26442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:25.316265 26441 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5221458556944d18125cfd85f217737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5221458556944d18125cfd85f217737" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 39781 } } }
I20260807 11:22:25.316646 26441 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:25.319835 26452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:25.327078 26452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:25.342635 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:25.315128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41599 (local address 127.2.223.190:39781)
0807 11:22:25.315406 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:25.315417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:25.319718 (+ 4301us) server_negotiation.cc:407] Connection header received
0807 11:22:25.320000 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:25.320013 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:25.320139 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:25.320290 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:25.322943 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:25.324176 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:25.326352 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:25.327168 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:25.327298 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:25.330436 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:25.330441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:25.330445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:25.330553 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:25.336128 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:25.337605 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:25.337614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:25.337621 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:25.337895 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:25.338691 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:25.338700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:25.338706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:25.339360 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:25.339742 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:25.341616 (+ 1874us) server_negotiation.cc:299] Negotiation successful
0807 11:22:25.342006 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:22:25.349954 26452 catalog_manager.cc:1347] Generated new cluster ID: 1352609fe4b848c190c618c1b6a73cc5
I20260807 11:22:25.350232 26452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:25.399961 26452 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:25.401549 26452 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:25.413753 26452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5221458556944d18125cfd85f217737: Generated new TSK 0
I20260807 11:22:25.414925 26452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:25.428268 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26364
I20260807 11:22:25.429692 26430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:25.529907 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26364
2026-08-07T11:22:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:25 dist-test-slave-5l3k krb5kdc[26469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:25 dist-test-slave-5l3k krb5kdc[26469](info): set up 2 sockets
Aug 07 11:22:25 dist-test-slave-5l3k krb5kdc[26469](info): commencing operation
krb5kdc: starting...
W20260807 11:22:26.748880 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:27 dist-test-slave-5l3k krb5kdc[26469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:27.170406 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40433
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:40433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:27.551028 26484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:27.551458 26484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:27.551656 26484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:27.559495 26484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:27.559701 26484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:27.559823 26484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:27.559942 26484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:27.560026 26484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:27.570796 26484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:40433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:27.572512 26484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:27.574246 26484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:27.588392 26490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:27.588358 26492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:27.588485 26489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:27.592345 26484 server_base.cc:1034] running on GCE node
I20260807 11:22:27.593371 26484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:27.595222 26484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:27.596493 26484 hybrid_clock.cc:648] HybridClock initialized: now 1786101747596455 us; error 49 us; skew 500 ppm
I20260807 11:22:27.596932 26484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:27 dist-test-slave-5l3k krb5kdc[26469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:27.600696 26484 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:27.602694 26484 webserver.cc:460] Webserver started at http://127.2.223.190:37397/ using document root <none> and password file <none>
I20260807 11:22:27.603499 26484 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:27.603673 26484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:27.604054 26484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:27.607717 26484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "b811111e94a34fe995cdbd7ce2bf7742"
format_stamp: "Formatted at 2026-08-07 11:22:27 on dist-test-slave-5l3k"
server_key: "2586e95d63f3199ebdc689cf7832a550"
server_key_iv: "daf53e7eb9a53661ffa3efd131b2000b"
server_key_version: "encryptionkey@0"
I20260807 11:22:27.608713 26484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "b811111e94a34fe995cdbd7ce2bf7742"
format_stamp: "Formatted at 2026-08-07 11:22:27 on dist-test-slave-5l3k"
server_key: "2586e95d63f3199ebdc689cf7832a550"
server_key_iv: "daf53e7eb9a53661ffa3efd131b2000b"
server_key_version: "encryptionkey@0"
I20260807 11:22:27.615415 26484 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:22:27.619832 26500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:27.622017 26484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260807 11:22:27.622375 26484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "b811111e94a34fe995cdbd7ce2bf7742"
format_stamp: "Formatted at 2026-08-07 11:22:27 on dist-test-slave-5l3k"
server_key: "2586e95d63f3199ebdc689cf7832a550"
server_key_iv: "daf53e7eb9a53661ffa3efd131b2000b"
server_key_version: "encryptionkey@0"
I20260807 11:22:27.622725 26484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:27.640204 26484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:27.645555 26484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:27.645920 26484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:27.666996 26484 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:40433
I20260807 11:22:27.666994 26556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:40433 every 8 connection(s)
I20260807 11:22:27.668728 26484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:27.673223 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:27.675850 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26484
I20260807 11:22:27.676075 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:27.676662 2942 external_mini_cluster.cc:1428] Setting key 0facc37749d933b497eca3e552188f7a
I20260807 11:22:27.687342 26557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742: Bootstrap starting.
I20260807 11:22:27.691488 26557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:27.692483 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:27.679056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49793 (local address 127.2.223.190:40433)
0807 11:22:27.680090 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:27.680104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:27.680162 (+ 58us) server_negotiation.cc:407] Connection header received
0807 11:22:27.682070 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:27.682100 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:27.682693 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:27.683161 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:27.683829 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:27.685356 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:27.686993 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:27.687897 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:27.688154 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:27.689568 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:27.689577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:27.689596 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:27.690243 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:27.690337 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:27.690367 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:27.690546 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:27.690867 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 11:22:27.691198 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":468,"threads_started":1}
I20260807 11:22:27.692914 26557 log.cc:824] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:27.696616 26557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742: No bootstrap required, opened a new log
I20260807 11:22:27.697158 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:27.705709 26557 raft_consensus.cc:348] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } }
I20260807 11:22:27.706247 26557 raft_consensus.cc:374] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:27.706388 26557 raft_consensus.cc:729] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811111e94a34fe995cdbd7ce2bf7742, State: Initialized, Role: FOLLOWER
I20260807 11:22:27.707118 26557 consensus_queue.cc:260] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } }
I20260807 11:22:27.707412 26557 raft_consensus.cc:388] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:27.707590 26557 raft_consensus.cc:482] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:27.707795 26557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:27.709254 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:27.702050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38939 (local address 127.2.223.190:40433)
0807 11:22:27.702288 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:27.702295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:27.702879 (+ 584us) server_negotiation.cc:407] Connection header received
0807 11:22:27.703137 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:27.703142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:27.703221 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:27.703351 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:27.703954 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:27.704748 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:27.706166 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:27.706904 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:27.707042 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:27.708415 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:27.708421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:27.708426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:27.708451 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:27.708497 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:27.708516 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:27.708640 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:27.708804 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 11:22:27.708921 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:22:27.710978 26557 raft_consensus.cc:504] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } }
I20260807 11:22:27.711630 26557 leader_election.cc:302] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811111e94a34fe995cdbd7ce2bf7742; no voters:
I20260807 11:22:27.712316 26557 leader_election.cc:288] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:27.712500 26567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:27.713327 26567 raft_consensus.cc:686] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [term 1 LEADER]: Becoming Leader. State: Replica: b811111e94a34fe995cdbd7ce2bf7742, State: Running, Role: LEADER
I20260807 11:22:27.713919 26567 consensus_queue.cc:237] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } }
I20260807 11:22:27.714896 26557 sys_catalog.cc:564] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:27.717013 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b811111e94a34fe995cdbd7ce2bf7742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } } }
I20260807 11:22:27.717509 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:27.718305 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b811111e94a34fe995cdbd7ce2bf7742. Latest consensus state: current_term: 1 leader_uuid: "b811111e94a34fe995cdbd7ce2bf7742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811111e94a34fe995cdbd7ce2bf7742" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 40433 } } }
I20260807 11:22:27.718798 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:27.723567 26572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:27.731858 26572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:27.742941 26572 catalog_manager.cc:1347] Generated new cluster ID: 3b44fa381f9f4c50810567a53ceae437
I20260807 11:22:27.743089 26572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:27.765522 26572 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:27.767314 26572 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:27.776445 26572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b811111e94a34fe995cdbd7ce2bf7742: Generated new TSK 0
I20260807 11:22:27.777556 26572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:27.792760 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26484
W20260807 11:22:27.794160 26551 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49793
W20260807 11:22:27.795066 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26484: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:27.795271 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26484
2026-08-07T11:22:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:27 dist-test-slave-5l3k krb5kdc[26589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:27 dist-test-slave-5l3k krb5kdc[26589](info): set up 2 sockets
Aug 07 11:22:27 dist-test-slave-5l3k krb5kdc[26589](info): commencing operation
krb5kdc: starting...
W20260807 11:22:29.003945 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:29 dist-test-slave-5l3k krb5kdc[26589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:29.418965 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41503
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:41503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:29.805104 26603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:29.805536 26603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:29.805722 26603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:29.813894 26603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:29.814123 26603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:29.814224 26603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:29.814322 26603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:29.814393 26603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:29.825388 26603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:41503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:41503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:29.827100 26603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:29.828668 26603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:29.843055 26612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:29.843124 26608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:29.843081 26609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:29.844069 26603 server_base.cc:1034] running on GCE node
I20260807 11:22:29.845017 26603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:29.846807 26603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:29.848115 26603 hybrid_clock.cc:648] HybridClock initialized: now 1786101749848008 us; error 33 us; skew 500 ppm
I20260807 11:22:29.848546 26603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:29 dist-test-slave-5l3k krb5kdc[26589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101749, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:29.852180 26603 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:29.854012 26603 webserver.cc:460] Webserver started at http://127.2.223.190:43235/ using document root <none> and password file <none>
I20260807 11:22:29.854934 26603 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:29.855115 26603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:29.855489 26603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:29.859169 26603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "04e9e6474bd542cf908a9f4a0463a211"
format_stamp: "Formatted at 2026-08-07 11:22:29 on dist-test-slave-5l3k"
server_key: "8306b2b40ee4c994a6d6447af168eed0"
server_key_iv: "a5852a455b00efcc1cccae4abffc9711"
server_key_version: "encryptionkey@0"
I20260807 11:22:29.860150 26603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "04e9e6474bd542cf908a9f4a0463a211"
format_stamp: "Formatted at 2026-08-07 11:22:29 on dist-test-slave-5l3k"
server_key: "8306b2b40ee4c994a6d6447af168eed0"
server_key_iv: "a5852a455b00efcc1cccae4abffc9711"
server_key_version: "encryptionkey@0"
I20260807 11:22:29.866407 26603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:22:29.870803 26620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:29.872520 26603 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:22:29.872776 26603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "04e9e6474bd542cf908a9f4a0463a211"
format_stamp: "Formatted at 2026-08-07 11:22:29 on dist-test-slave-5l3k"
server_key: "8306b2b40ee4c994a6d6447af168eed0"
server_key_iv: "a5852a455b00efcc1cccae4abffc9711"
server_key_version: "encryptionkey@0"
I20260807 11:22:29.873034 26603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:29.880050 26603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:29.885330 26603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:29.885728 26603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:29.907054 26603 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:41503
I20260807 11:22:29.907094 26679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:41503 every 8 connection(s)
I20260807 11:22:29.908834 26603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:29.911795 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26603
I20260807 11:22:29.912026 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:29.912528 2942 external_mini_cluster.cc:1428] Setting key a92c989e24cee3be8cfc6e50db42c4fa
I20260807 11:22:29.913754 26680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:29.921926 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:29.914666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41251 (local address 127.2.223.190:41503)
0807 11:22:29.915608 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:29.915623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:29.915670 (+ 47us) server_negotiation.cc:407] Connection header received
0807 11:22:29.917474 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:29.917501 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:29.917966 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:29.918750 (+ 784us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:29.920025 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:29.920035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:29.920051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:29.920067 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:29.920157 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:29.920178 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:29.920461 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:29.920632 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 11:22:29.920865 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":419,"threads_started":1}
I20260807 11:22:29.926461 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:29.928800 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211: Bootstrap starting.
I20260807 11:22:29.932920 26680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:29.934366 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:29.931489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42249 (local address 127.2.223.190:41503)
0807 11:22:29.931716 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:29.931723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:29.932186 (+ 463us) server_negotiation.cc:407] Connection header received
0807 11:22:29.932353 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:29.932358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:29.932434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:29.932569 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:29.933627 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:29.933632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:29.933637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:22:29.933658 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:29.933714 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:22:29.933737 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:29.933871 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:29.933968 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 11:22:29.934048 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:22:29.934881 26680 log.cc:824] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:29.938675 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211: No bootstrap required, opened a new log
I20260807 11:22:29.947621 26680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } }
I20260807 11:22:29.948154 26680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:22:29.948246 26680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e9e6474bd542cf908a9f4a0463a211, State: Initialized, Role: FOLLOWER
I20260807 11:22:29.948910 26680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } }
I20260807 11:22:29.949180 26680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:22:29.949299 26680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:22:29.949443 26680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:29.952052 26680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } }
I20260807 11:22:29.952663 26680 leader_election.cc:302] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e9e6474bd542cf908a9f4a0463a211; no voters:
I20260807 11:22:29.953392 26680 leader_election.cc:288] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:22:29.953538 26691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:29.955056 26691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [term 1 LEADER]: Becoming Leader. State: Replica: 04e9e6474bd542cf908a9f4a0463a211, State: Running, Role: LEADER
I20260807 11:22:29.955772 26691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } }
I20260807 11:22:29.956013 26680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:29.959234 26695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e9e6474bd542cf908a9f4a0463a211. Latest consensus state: current_term: 1 leader_uuid: "04e9e6474bd542cf908a9f4a0463a211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } } }
I20260807 11:22:29.959821 26695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:29.959723 26694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e9e6474bd542cf908a9f4a0463a211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e9e6474bd542cf908a9f4a0463a211" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 41503 } } }
I20260807 11:22:29.960014 26694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211 [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:29.962304 26702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:29.967864 26702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:29.978700 26702 catalog_manager.cc:1347] Generated new cluster ID: e2d24618b12c40cb8d7e03eef20ca9d8
I20260807 11:22:29.978858 26702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:29.994843 26702 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:29.995406 26702 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:30.008630 26702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e9e6474bd542cf908a9f4a0463a211: Generated new TSK 0
I20260807 11:22:30.009670 26702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:22:30.020856 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26603
W20260807 11:22:30.022271 26675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41251
W20260807 11:22:30.023197 2942 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26603: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:22:30.023435 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26603
2026-08-07T11:22:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2227 ms)
[----------] 18 tests from AuthTokenIssuingTest (41561 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:30 dist-test-slave-5l3k krb5kdc[26715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:30 dist-test-slave-5l3k krb5kdc[26715](info): set up 2 sockets
Aug 07 11:22:30 dist-test-slave-5l3k krb5kdc[26715](info): commencing operation
krb5kdc: starting...
W20260807 11:22:31.233631 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.003s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:31 dist-test-slave-5l3k krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:31.654609 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35967
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:32.051380 26730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:32.051966 26730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:32.052109 26730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:32.052258 26730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:32.064378 26730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:32.064538 26730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:32.064653 26730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:32.064751 26730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:32.064853 26730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:32.081060 26730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:32.083232 26730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:32.085135 26730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:32.103092 26735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:32.103003 26736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:32.103039 26738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:32.104907 26730 server_base.cc:1034] running on GCE node
I20260807 11:22:32.105922 26730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:32.107867 26730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:32.109203 26730 hybrid_clock.cc:648] HybridClock initialized: now 1786101752109108 us; error 70 us; skew 500 ppm
I20260807 11:22:32.109679 26730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:32.113337 26730 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:32.115320 26730 webserver.cc:460] Webserver started at http://127.2.223.190:37015/ using document root <none> and password file <none>
I20260807 11:22:32.116154 26730 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:32.116312 26730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:32.116691 26730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:32.120349 26730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "4be3a4a573b94a72889c16b1d66cbaab"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "8ee5d545352a98d345b703ede4e37300"
server_key_iv: "8c18e911451c87b44bd6008fe7693112"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.121338 26730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "4be3a4a573b94a72889c16b1d66cbaab"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "8ee5d545352a98d345b703ede4e37300"
server_key_iv: "8c18e911451c87b44bd6008fe7693112"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.127722 26730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:22:32.132182 26745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:32.134003 26730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:22:32.134277 26730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "4be3a4a573b94a72889c16b1d66cbaab"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "8ee5d545352a98d345b703ede4e37300"
server_key_iv: "8c18e911451c87b44bd6008fe7693112"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.134591 26730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:32.149149 26730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:32.154685 26730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:32.155095 26730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:32.177433 26730 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:35967
I20260807 11:22:32.177452 26800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:35967 every 8 connection(s)
I20260807 11:22:32.179236 26730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:32.183954 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:32.188817 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26730
I20260807 11:22:32.189095 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:32.190066 2942 external_mini_cluster.cc:1428] Setting key a4cfff6f1f00b2f96f9d29c7cec9592a
I20260807 11:22:32.194623 26801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:32.224354 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.199613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35967 (local address 127.0.0.1:58966)
0807 11:22:32.200712 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.200778 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:32.203223 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:32.204306 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:32.204313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:32.205155 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:32.205533 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.205554 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.207418 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.207422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.208615 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.208641 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.210225 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.210232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.210430 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.211356 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:32.211385 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:32.213954 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:32.219356 (+ 5402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.219369 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.219382 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.219894 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.220590 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.220594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.220597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.220786 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.221964 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:32.221971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:32.222197 (+ 226us) client_negotiation.cc:769] Sending connection context
0807 11:22:32.222448 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 11:22:32.223052 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":8,"thread_start_us":469,"threads_started":1}
I20260807 11:22:32.224406 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.199614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.2.223.190:35967)
0807 11:22:32.201738 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:32.201748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:32.201814 (+ 66us) server_negotiation.cc:407] Connection header received
0807 11:22:32.203714 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:32.203721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:32.204022 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:32.204277 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:32.205796 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.207210 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.208802 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.209850 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.210116 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.214330 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:32.214353 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:32.214377 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:32.214432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:32.219103 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.220080 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.220094 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.220100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.220393 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.220923 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.220930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.220936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.221785 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:32.222037 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:32.222697 (+ 660us) server_negotiation.cc:299] Negotiation successful
0807 11:22:32.223275 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":1867,"thread_start_us":1597,"threads_started":1}
I20260807 11:22:32.228494 26801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:32.229918 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.229356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55790)
0807 11:22:32.229605 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.229758 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260807 11:22:32.230233 26748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
I20260807 11:22:32.232282 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.231809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55796)
0807 11:22:32.232029 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.232109 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 11:22:32.232513 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:39543: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.223.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
I20260807 11:22:32.281703 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:39543
--webserver_interface=127.2.223.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:22:32.281935 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } attempt: 1
I20260807 11:22:32.283533 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.282873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55806)
0807 11:22:32.283152 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.283327 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:22:32.284917 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.284382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55822)
0807 11:22:32.284594 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.284703 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 11:22:32.285209 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:39543: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
I20260807 11:22:32.353606 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } attempt: 2
I20260807 11:22:32.355119 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.354467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55828)
0807 11:22:32.354838 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.354959 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 11:22:32.356314 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.355897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55836)
0807 11:22:32.356076 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.356170 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 11:22:32.356528 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:39543: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
I20260807 11:22:32.447804 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } attempt: 3
I20260807 11:22:32.449133 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.448654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55842)
0807 11:22:32.448878 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.448981 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 11:22:32.450328 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.449918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55852)
0807 11:22:32.450086 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.450174 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 11:22:32.450567 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:39543: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
I20260807 11:22:32.594082 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } attempt: 4
I20260807 11:22:32.595520 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.594925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55862)
0807 11:22:32.595274 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.595371 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260807 11:22:32.596877 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.596414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55878)
0807 11:22:32.596606 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.596721 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 11:22:32.597105 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:39543: Network error: Client connection negotiation failed: client connection to 127.2.223.189:39543: connect: Connection refused (error 111)
W20260807 11:22:32.674197 26809 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:32.674710 26809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:32.674860 26809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:32.675009 26809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:32.683005 26809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:32.683202 26809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:32.683322 26809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:32.683421 26809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:32.683523 26809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:32.695317 26809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.223.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:32.696933 26809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:32.698599 26809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:32.712090 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:32.712250 26815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:32.712116 26814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:32.716440 26809 server_base.cc:1034] running on GCE node
I20260807 11:22:32.717535 26809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:32.719498 26809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:32.720834 26809 hybrid_clock.cc:648] HybridClock initialized: now 1786101752720766 us; error 54 us; skew 500 ppm
I20260807 11:22:32.721259 26809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:32.725185 26809 init.cc:377] Logged in from keytab as kudu/127.2.223.189@KRBTEST.COM (short username kudu)
I20260807 11:22:32.727245 26809 webserver.cc:460] Webserver started at http://127.2.223.189:33779/ using document root <none> and password file <none>
I20260807 11:22:32.728168 26809 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:32.728331 26809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:32.728729 26809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:32.732369 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data/instance:
uuid: "886ffc4293234e8086384cd515f0e683"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "7f7ab51f4be5162aba1c0544bd29e299"
server_key_iv: "4148d3fc44b805f6733f60bde0900d8e"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.733470 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal/instance:
uuid: "886ffc4293234e8086384cd515f0e683"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "7f7ab51f4be5162aba1c0544bd29e299"
server_key_iv: "4148d3fc44b805f6733f60bde0900d8e"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.740234 26809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 11:22:32.744817 26824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:32.746613 26809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260807 11:22:32.746888 26809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "886ffc4293234e8086384cd515f0e683"
format_stamp: "Formatted at 2026-08-07 11:22:32 on dist-test-slave-5l3k"
server_key: "7f7ab51f4be5162aba1c0544bd29e299"
server_key_iv: "4148d3fc44b805f6733f60bde0900d8e"
server_key_version: "encryptionkey@0"
I20260807 11:22:32.747188 26809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:32.769476 26809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:32.774303 26809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:32.774694 26809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:32.796367 26809 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.189:39543
I20260807 11:22:32.796382 26880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.189:39543 every 8 connection(s)
I20260807 11:22:32.798058 26809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
I20260807 11:22:32.802980 26881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:32.805176 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26809
I20260807 11:22:32.805410 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-1/wal/instance
I20260807 11:22:32.805992 2942 external_mini_cluster.cc:1428] Setting key 55509f3561cf3c0090362f6e9703c8b3
I20260807 11:22:32.814165 26881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:32.843097 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.818375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.2.223.190:35967)
0807 11:22:32.819007 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:32.819014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:32.820473 (+ 1459us) server_negotiation.cc:407] Connection header received
0807 11:22:32.823933 (+ 3460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:32.823938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:32.824025 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:32.824140 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:32.826532 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.827810 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.829672 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.830532 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.830778 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.835429 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:32.835436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:32.835440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:32.835498 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:32.838782 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.839951 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.839962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.839965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.840165 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.840695 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.840701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.840705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.841028 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:32.841217 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:32.842199 (+ 982us) server_negotiation.cc:299] Negotiation successful
0807 11:22:32.842546 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":327,"threads_started":1}
I20260807 11:22:32.845291 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.819196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35967 (local address 127.0.0.1:58972)
0807 11:22:32.820214 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.820289 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:32.823493 (+ 3204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:32.824508 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:32.824525 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:32.825445 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:32.826164 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.826187 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.828047 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.828051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.829433 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.829450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.830871 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.830877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.831127 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.832013 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:32.832041 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:32.835031 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:32.839090 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.839100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.839119 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.839783 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.840316 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.840322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.840326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.840561 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.841203 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:32.841213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:32.841678 (+ 465us) client_negotiation.cc:769] Sending connection context
0807 11:22:32.843274 (+ 1596us) client_negotiation.cc:240] Negotiation successful
0807 11:22:32.843869 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":363,"threads_started":1}
WARNING: no policy specified for kudu/127.2.223.188@KRBTEST.COM; defaulting to no policy
I20260807 11:22:32.847535 26881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.188@KRBTEST.COM" created.
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:32.874311 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.849101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.2.223.189:39543)
0807 11:22:32.854616 (+ 5515us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:32.854630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:32.854681 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:22:32.854777 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:32.854790 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:32.855137 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:32.855359 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:32.856777 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.858607 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.860585 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.861584 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.861746 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.866510 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:32.866534 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:32.866559 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:32.866613 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:32.870645 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.871756 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.871764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.871766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.871909 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.872393 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.872396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.872398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.872803 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:32.872997 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:32.873570 (+ 573us) server_negotiation.cc:299] Negotiation successful
0807 11:22:32.873962 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":5206,"thread_start_us":5011,"threads_started":1}
I20260807 11:22:32.874359 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.848585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55886)
0807 11:22:32.848804 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.848831 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:32.848992 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:32.855667 (+ 6675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:32.855674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:32.856282 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:32.856531 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.856547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.858848 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.858856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.860364 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.860386 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.861810 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.861819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.862107 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.862918 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:32.862949 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:32.866192 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:32.870859 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.870868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.870875 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.871569 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.872066 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.872074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.872080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.872274 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.873017 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:32.873025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:32.873243 (+ 218us) client_negotiation.cc:769] Sending connection context
0807 11:22:32.873423 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 11:22:32.873823 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 11:22:32.877560 26881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:32.879134 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.878346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41536)
0807 11:22:32.878639 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.878859 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260807 11:22:32.879468 26829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:32.881510 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.880952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41542)
0807 11:22:32.881188 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.881318 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260807 11:22:32.881821 26881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:32.896373 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
I20260807 11:22:32.900074 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:34595
--webserver_interface=127.2.223.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:22:32 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:32.913609 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.897293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.2.223.189:39543)
0807 11:22:32.897512 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:32.897517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:32.897539 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:22:32.897716 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:32.897720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:32.897808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:32.897904 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:32.899120 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.900246 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.902415 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.903066 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.903202 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.907461 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:32.907467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:32.907470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:32.907511 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:32.910585 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.911423 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.911430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.911433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.911587 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:32.912030 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:32.912036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:32.912041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:32.912372 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:32.912545 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:32.912986 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 11:22:32.913250 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 11:22:32.913623 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.897197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55898)
0807 11:22:32.897416 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.897441 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:32.897563 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:32.897966 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:32.897970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:32.898593 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:32.898901 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.898919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.900466 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.900474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.902115 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:32.902143 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:32.903281 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:32.903292 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:32.903603 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:32.904346 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:32.904378 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:32.907185 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:32.910778 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.910784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.910788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.911261 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.911726 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:32.911731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:32.911735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:32.911901 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:32.912510 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:32.912515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:32.912695 (+ 180us) client_negotiation.cc:769] Sending connection context
0807 11:22:32.912857 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 11:22:32.913239 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 11:22:32.914788 26801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:32.915935 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.915486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41546)
0807 11:22:32.915674 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.915775 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 11:22:32.917063 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.916674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41548)
0807 11:22:32.916837 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.916945 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:22:32.917356 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:32.931282 26881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 1
I20260807 11:22:32.932688 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.932162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41562)
0807 11:22:32.932388 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.932505 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 11:22:32.933859 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.933444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41574)
0807 11:22:32.933613 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.933724 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:22:32.934159 26881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:32.968608 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 1
I20260807 11:22:32.970005 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.969470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41590)
0807 11:22:32.969713 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.969816 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 11:22:32.971323 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:32.970911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41600)
0807 11:22:32.971094 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:32.971193 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 11:22:32.971561 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:33.002398 26881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 2
I20260807 11:22:33.003938 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.003370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41604)
0807 11:22:33.003576 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.003702 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 11:22:33.005810 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.005349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41620)
0807 11:22:33.005542 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.005649 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 11:22:33.006121 26881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:33.039816 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 2
I20260807 11:22:33.041145 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.040596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41630)
0807 11:22:33.040859 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.040960 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 11:22:33.042356 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.041935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41636)
0807 11:22:33.042149 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.042236 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 11:22:33.042623 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:33.097352 26881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 3
I20260807 11:22:33.098774 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.098190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41642)
0807 11:22:33.098431 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.098608 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 11:22:33.100023 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.099562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41652)
0807 11:22:33.099811 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.099894 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260807 11:22:33.100328 26881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:33.148874 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 3
I20260807 11:22:33.150161 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.149630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41666)
0807 11:22:33.149857 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.149962 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 11:22:33.151471 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.151015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41670)
0807 11:22:33.151219 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.151354 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 11:22:33.151729 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
I20260807 11:22:33.243587 26881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 4
I20260807 11:22:33.244931 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.244435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41686)
0807 11:22:33.244663 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.244763 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:22:33.246030 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.245671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41700)
0807 11:22:33.245830 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.245913 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:22:33.246248 26881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
W20260807 11:22:33.300999 26890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:33.301481 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:33.301626 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:33.301771 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:33.309959 26890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:33.310143 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:33.310272 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:33.310367 26890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:33.310525 26890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:33.321547 26890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:35967,127.2.223.189:39543,127.2.223.188:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.223.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:33.323393 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:33.324972 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 11:22:33.328953 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 4
I20260807 11:22:33.330322 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.329785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41712)
0807 11:22:33.330006 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.330116 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:22:33.331650 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.331188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41722)
0807 11:22:33.331376 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.331472 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 11:22:33.332033 26801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:34595: Network error: Client connection negotiation failed: client connection to 127.2.223.188:34595: connect: Connection refused (error 111)
W20260807 11:22:33.339313 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:33.339607 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:33.339839 26899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:33.340997 26890 server_base.cc:1034] running on GCE node
I20260807 11:22:33.341977 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:33.343909 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:33.345248 26890 hybrid_clock.cc:648] HybridClock initialized: now 1786101753345176 us; error 54 us; skew 500 ppm
I20260807 11:22:33.345708 26890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:33.349493 26890 init.cc:377] Logged in from keytab as kudu/127.2.223.188@KRBTEST.COM (short username kudu)
I20260807 11:22:33.351545 26890 webserver.cc:460] Webserver started at http://127.2.223.188:33021/ using document root <none> and password file <none>
I20260807 11:22:33.352443 26890 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:33.352615 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:33.353071 26890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:33.356868 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data/instance:
uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb"
format_stamp: "Formatted at 2026-08-07 11:22:33 on dist-test-slave-5l3k"
server_key: "94f5aaad0e4fa5704ed2954e960b624d"
server_key_iv: "d1e982f637a802de5e563d36a21cd153"
server_key_version: "encryptionkey@0"
I20260807 11:22:33.358191 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal/instance:
uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb"
format_stamp: "Formatted at 2026-08-07 11:22:33 on dist-test-slave-5l3k"
server_key: "94f5aaad0e4fa5704ed2954e960b624d"
server_key_iv: "d1e982f637a802de5e563d36a21cd153"
server_key_version: "encryptionkey@0"
I20260807 11:22:33.365029 26890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:22:33.369311 26906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:33.371153 26890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260807 11:22:33.371461 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb"
format_stamp: "Formatted at 2026-08-07 11:22:33 on dist-test-slave-5l3k"
server_key: "94f5aaad0e4fa5704ed2954e960b624d"
server_key_iv: "d1e982f637a802de5e563d36a21cd153"
server_key_version: "encryptionkey@0"
I20260807 11:22:33.371757 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:33.392319 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:33.397050 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:33.397405 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:33.419248 26890 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.188:34595
I20260807 11:22:33.419297 26960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.188:34595 every 8 connection(s)
I20260807 11:22:33.420831 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
I20260807 11:22:33.425144 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:33.425273 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 26890
I20260807 11:22:33.425515 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786101369622807-2942-0/minicluster-data/master-2/wal/instance
I20260807 11:22:33.425985 2942 external_mini_cluster.cc:1428] Setting key bedf808724658f5a64f8bf64bc214867
I20260807 11:22:33.436167 26962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:33.453262 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.428357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.2.223.188:34595)
0807 11:22:33.429257 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.429271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.429318 (+ 47us) server_negotiation.cc:407] Connection header received
0807 11:22:33.431048 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.431074 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.431545 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.431929 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.433888 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.435328 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.437325 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.438184 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.438340 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.443787 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.443796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.443809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.443901 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.448341 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.449165 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.449177 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.449191 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.449375 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.449894 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.449901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.449907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.450635 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.451088 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.451530 (+ 442us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.452080 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":394,"threads_started":1}
I20260807 11:22:33.457898 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.439795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:35967 (local address 127.0.0.1:58976)
0807 11:22:33.440843 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.440892 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:33.441078 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:33.441533 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:33.441539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:33.442195 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:33.442557 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.442578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.444167 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.444173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.446054 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.446076 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.447025 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.447032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.447213 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.447885 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:33.447922 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:33.450629 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:33.454423 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.454435 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.454442 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.455068 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.455599 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.455604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.455607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.455797 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.456481 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:33.456488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:33.456682 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 11:22:33.456909 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 11:22:33.457322 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":587,"threads_started":1}
I20260807 11:22:33.457924 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.439786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.2.223.190:35967)
0807 11:22:33.440395 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.440402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.441012 (+ 610us) server_negotiation.cc:407] Connection header received
0807 11:22:33.441218 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.441224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.441324 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.441461 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.442745 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.443986 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.446273 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.446800 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.446937 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.450862 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.450868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.450871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.450920 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.454257 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.455222 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.455228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.455231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.455420 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.455961 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.455968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.455972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.456291 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.456493 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.457135 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.457389 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":291,"threads_started":1}
I20260807 11:22:33.460139 26962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:33.476045 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.457132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.2.223.190:35967)
0807 11:22:33.458359 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.458363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.458384 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:22:33.458425 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.458428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.458558 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.458675 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.460689 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.461830 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.463285 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.463768 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.463926 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.468880 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.468892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.468900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.468967 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.472517 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.473630 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.473637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.473640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.473868 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.474336 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.474344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.474350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.474800 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.475125 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.475389 (+ 264us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.475603 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":754,"threads_started":1}
I20260807 11:22:33.476904 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.460943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:39543 (local address 127.0.0.1:55906)
0807 11:22:33.461177 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.461201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:33.461312 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:33.462170 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:33.462176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:33.462701 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:33.462890 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.462902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.464096 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.464105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.465099 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.465114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.465976 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.465984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.466223 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.466950 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:33.466973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:33.469469 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:33.473661 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.473667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.473670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.474300 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.474833 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.474841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.474848 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.475045 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.475716 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:33.475722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:33.475915 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 11:22:33.476122 (+ 207us) client_negotiation.cc:240] Negotiation successful
0807 11:22:33.476525 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 11:22:33.476949 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.461163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.2.223.189:39543)
0807 11:22:33.461756 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.461765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.461791 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:22:33.461867 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.461874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.461964 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.462135 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.463152 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.463904 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.465294 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.465771 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.465944 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.469749 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.469756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.469760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.469837 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.473460 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.474450 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.474455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.474457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.474696 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.475193 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.475199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.475202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.475552 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.475735 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.476267 (+ 532us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.476525 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":690,"threads_started":1}
I20260807 11:22:33.478129 26962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:33.496060 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.479168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.2.223.188:34595)
0807 11:22:33.479307 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.479311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.479327 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:22:33.479450 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.479454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.479506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.479581 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.480657 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.481769 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.483744 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.484438 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.484586 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.489107 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.489115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.489120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.489174 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.492791 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.493880 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.493885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.493889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.494073 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.494510 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.494515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.494518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.494838 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.495013 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.495383 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.495660 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 11:22:33.496208 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.478990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41738)
0807 11:22:33.479211 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.479232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:33.479342 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:33.479658 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:33.479663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:33.480200 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:33.480417 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.480432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.481982 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.481988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.483539 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.483560 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.484652 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.484659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.484912 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.485723 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:33.485748 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:33.488780 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:33.493064 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.493070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.493074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.493728 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.494204 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.494209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.494213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.494359 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.494981 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:33.494985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:33.495147 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 11:22:33.495301 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 11:22:33.495848 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":180}
I20260807 11:22:33.497480 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.478326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.2.223.189:39543)
0807 11:22:33.478554 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.478559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.478589 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:22:33.478634 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.478637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.478697 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.478812 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.480668 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.481772 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.483759 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.484438 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.484586 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.490301 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.490306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.490310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.490351 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.494085 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.495048 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.495054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.495056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.495210 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.495672 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.495677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.495679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.495974 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.496293 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.496759 (+ 466us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.497069 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:22:33.503603 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:33.503705 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: Bootstrap starting.
I20260807 11:22:33.507718 26962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:33.509272 26962 log.cc:824] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:33.506757 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4be3a4a573b94a72889c16b1d66cbaab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58984
I20260807 11:22:33.513104 26962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: No bootstrap required, opened a new log
I20260807 11:22:33.525122 26962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.525663 26962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b76f8a8ddaa4dce8d2a5ba2ce633acb, State: Initialized, Role: FOLLOWER
I20260807 11:22:33.526268 26962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.527513 26971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.527899 26971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.528965 26962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:22:33.540634 26982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:22:33.540834 26982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:22:33.545439 26881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 5
I20260807 11:22:33.537868 26890 master_runner.cc:186] Error getting master registration for 127.2.223.190:35967: OK, instance_id { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" instance_seqno: 1786101752171246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:22:33.550693 26890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:22:33.564574 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.546452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.2.223.188:34595)
0807 11:22:33.546750 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.546757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.546838 (+ 81us) server_negotiation.cc:407] Connection header received
0807 11:22:33.547089 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.547095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.547193 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.547305 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.548242 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.549544 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.551119 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.551979 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.552141 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.556829 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.556836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.556840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.556887 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.561124 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.562123 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.562129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.562133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.562296 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.562844 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.562850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.562853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.563247 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.563464 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.563915 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.564178 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:22:33.564579 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.546353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41750)
0807 11:22:33.546718 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.546740 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:33.546898 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:33.547354 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:33.547361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:33.547843 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:33.548036 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.548050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.549748 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.549754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.550929 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.550947 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.552173 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.552179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.552550 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.553345 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:33.553372 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:33.556514 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:33.561371 (+ 4857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.561378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.561382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.561917 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.562458 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.562464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.562468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.562693 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.563405 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:33.563411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:33.563606 (+ 195us) client_negotiation.cc:769] Sending connection context
0807 11:22:33.563767 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:22:33.564158 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260807 11:22:33.570169 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: Bootstrap starting.
I20260807 11:22:33.575052 26881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:33.576704 26881 log.cc:824] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:33.580350 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: No bootstrap required, opened a new log
I20260807 11:22:33.592810 26881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.593339 26881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886ffc4293234e8086384cd515f0e683, State: Initialized, Role: FOLLOWER
I20260807 11:22:33.593889 26881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.595118 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.595602 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.596503 26881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:33.609259 26801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } attempt: 5
W20260807 11:22:33.615090 26999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:22:33.615525 26999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 07 11:22:33 dist-test-slave-5l3k krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:22:33.609413 26809 master_runner.cc:186] Error getting master registration for 127.2.223.190:35967: OK, instance_id { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" instance_seqno: 1786101752171246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:22:33.622655 26809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:22:33.630936 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.610832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.2.223.188:34595)
0807 11:22:33.611033 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.611050 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.611564 (+ 514us) server_negotiation.cc:407] Connection header received
0807 11:22:33.611650 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.611655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.611750 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.611851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.613102 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.614357 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.616419 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.617171 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.617351 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.622120 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.622129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.622135 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.622187 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.626463 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.627617 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.627624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.627627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.627812 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.628340 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.628346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.628351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.628743 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.630038 (+ 1295us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.630187 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.630455 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:22:33.631256 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.611102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:34595 (local address 127.0.0.1:41752)
0807 11:22:33.611274 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:33.611296 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:33.611439 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:33.612085 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:33.612091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:33.612637 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:33.612879 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.612892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.614599 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.614605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.616180 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:33.616200 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.617379 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.617387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:33.617632 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.618511 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:33.618541 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:33.621814 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:33.626730 (+ 4916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.626737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.626742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.627429 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.627978 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:33.627984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:33.627988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:33.628192 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:33.628973 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:33.628979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:33.629182 (+ 203us) client_negotiation.cc:769] Sending connection context
0807 11:22:33.629327 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 11:22:33.629669 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 11:22:33.638324 26801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab: Bootstrap starting.
I20260807 11:22:33.643391 26801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:33.645156 26801 log.cc:824] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab: Log is configured to *not* fsync() on all Append() calls
W20260807 11:22:33.649966 26913 tablet.cc:2369] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:22:33.650106 26801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab: No bootstrap required, opened a new log
I20260807 11:22:33.659262 26801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.659963 26801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be3a4a573b94a72889c16b1d66cbaab, State: Initialized, Role: FOLLOWER
I20260807 11:22:33.660684 26801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.662395 27002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.663317 27002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.664000 26801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:33.664094 26773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:22:33.664324 26773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:33.668375 26773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.671407 26855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4be3a4a573b94a72889c16b1d66cbaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "886ffc4293234e8086384cd515f0e683"
I20260807 11:22:33.672031 26855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:33.677481 26855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4be3a4a573b94a72889c16b1d66cbaab in term 1.
I20260807 11:22:33.678719 26748 leader_election.cc:302] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4be3a4a573b94a72889c16b1d66cbaab, 886ffc4293234e8086384cd515f0e683; no voters:
I20260807 11:22:33.680680 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4be3a4a573b94a72889c16b1d66cbaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb"
I20260807 11:22:33.681320 26936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:33.685352 26936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4be3a4a573b94a72889c16b1d66cbaab in term 1.
I20260807 11:22:33.686659 26773 leader_election.cc:288] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [CANDIDATE]: Term 1 election: Requested vote from peers 886ffc4293234e8086384cd515f0e683 (127.2.223.189:39543), 4b76f8a8ddaa4dce8d2a5ba2ce633acb (127.2.223.188:34595)
I20260807 11:22:33.694245 27002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:33.703917 27002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [term 1 LEADER]: Becoming Leader. State: Replica: 4be3a4a573b94a72889c16b1d66cbaab, State: Running, Role: LEADER
I20260807 11:22:33.705866 27002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } }
I20260807 11:22:33.717514 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be3a4a573b94a72889c16b1d66cbaab. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.718225 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:33.723465 27018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:33.742775 27018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:22:33.749884 27025 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 11:22:33.777850 26832 tablet.cc:2369] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:22:33.777853 26855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [term 1 FOLLOWER]: Refusing update from remote peer 4be3a4a573b94a72889c16b1d66cbaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:22:33.779165 26936 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [term 1 FOLLOWER]: Refusing update from remote peer 4be3a4a573b94a72889c16b1d66cbaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:22:33.780580 27002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:22:33.782435 27002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:22:33.810969 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be3a4a573b94a72889c16b1d66cbaab. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.811331 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.814409 27028 mvcc.cc:205] Tried to move back new op lower bound from 7315872783456952320 to 7315872783199846400. Current Snapshot: MvccSnapshot[applied={T|T < 7315872783456952320}]
I20260807 11:22:33.825052 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.825407 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.836097 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.836519 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:33.855916 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.745151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.2.223.189:39543)
0807 11:22:33.745431 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.745440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.749948 (+ 4508us) server_negotiation.cc:407] Connection header received
0807 11:22:33.752582 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.752589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.752695 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.752776 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.770689 (+ 17913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.771994 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.793751 (+ 21757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.794558 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.794688 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.837113 (+ 42425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.837125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.837131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.837227 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.842224 (+ 4997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.852488 (+ 10264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.852496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.852499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.852785 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.853363 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.853371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.853374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.853956 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.854269 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.855058 (+ 789us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.855383 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:22:33.856379 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.730015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.2.223.190:35967)
0807 11:22:33.730274 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.730281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.744171 (+ 13890us) server_negotiation.cc:407] Connection header received
0807 11:22:33.746015 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.746022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.746148 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.746236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.763195 (+ 16959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.764403 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.767892 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.768704 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.768841 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.809987 (+ 41146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.810000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.810006 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.810063 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.819052 (+ 8989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.843585 (+ 24533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.843595 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.843599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.843826 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.853422 (+ 9596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.853432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.853436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.853904 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.854269 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.855407 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.855715 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:22:33.858562 27018 catalog_manager.cc:1347] Generated new cluster ID: d15231c609e340b4964f77cdd85e7d02
I20260807 11:22:33.858703 27018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:33.866235 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:33.750660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.2.223.188:34595)
0807 11:22:33.750882 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:33.750889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:33.752182 (+ 1293us) server_negotiation.cc:407] Connection header received
0807 11:22:33.760304 (+ 8122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:33.760314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:33.760424 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:33.760513 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:33.772259 (+ 11746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.773433 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.797392 (+ 23959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:33.798221 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:33.798365 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:33.844957 (+ 46592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:33.844967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:33.844973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:33.845062 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:33.849676 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.858928 (+ 9252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.858937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.858941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.859569 (+ 628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:33.861924 (+ 2355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:33.861931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:33.861935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:33.862548 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:33.862874 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:33.865298 (+ 2424us) server_negotiation.cc:299] Negotiation successful
0807 11:22:33.865719 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:22:33.880331 26971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be3a4a573b94a72889c16b1d66cbaab. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.880774 26971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.885715 27002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.886150 27002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:33.892793 26971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4be3a4a573b94a72889c16b1d66cbaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be3a4a573b94a72889c16b1d66cbaab" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 35967 } } peers { permanent_uuid: "886ffc4293234e8086384cd515f0e683" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 39543 } } peers { permanent_uuid: "4b76f8a8ddaa4dce8d2a5ba2ce633acb" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 34595 } } }
I20260807 11:22:33.893107 26971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:33.917241 27018 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:33.919435 27018 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:33.950935 27018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be3a4a573b94a72889c16b1d66cbaab: Generated new TSK 0
I20260807 11:22:33.952476 27018 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:22:34.117077 26968 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.104533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.2.223.190:35967)
0807 11:22:34.104811 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.104818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.113147 (+ 8329us) server_negotiation.cc:407] Connection header received
0807 11:22:34.114963 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.114972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.115085 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.115181 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.116660 (+ 1479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59014: BlockingRecv error: recv got EOF from 127.0.0.1:59014 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260807 11:22:34.121973 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.113945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.2.223.189:39543)
0807 11:22:34.114220 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.114226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.119093 (+ 4867us) server_negotiation.cc:407] Connection header received
0807 11:22:34.120194 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.120202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.120319 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.120420 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.121680 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55942: BlockingRecv error: recv got EOF from 127.0.0.1:55942 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 11:22:34.126295 26965 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.123215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.2.223.188:34595)
0807 11:22:34.123424 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.123431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.124455 (+ 1024us) server_negotiation.cc:407] Connection header received
0807 11:22:34.124739 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.124746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.124836 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.124914 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.126014 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41782: BlockingRecv error: recv got EOF from 127.0.0.1:41782 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 11:22:34.173660 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.162678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.2.223.189:39543)
0807 11:22:34.162904 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.162911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.171043 (+ 8132us) server_negotiation.cc:407] Connection header received
0807 11:22:34.171724 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.171732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.171890 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.171988 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.173369 (+ 1381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55948: BlockingRecv error: recv got EOF from 127.0.0.1:55948 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260807 11:22:34.216344 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.206140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.2.223.189:39543)
0807 11:22:34.206356 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.206363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.214027 (+ 7664us) server_negotiation.cc:407] Connection header received
0807 11:22:34.214707 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.214714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.214822 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.214900 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.216110 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55962: BlockingRecv error: recv got EOF from 127.0.0.1:55962 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:22:34.241099 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.235957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.2.223.189:39543)
0807 11:22:34.236210 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.236220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.239376 (+ 3156us) server_negotiation.cc:407] Connection header received
0807 11:22:34.239642 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.239646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.239727 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.239862 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.240793 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55970: BlockingRecv error: recv got EOF from 127.0.0.1:55970 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:22:34.256412 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.253463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.2.223.189:39543)
0807 11:22:34.253819 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.253826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.254502 (+ 676us) server_negotiation.cc:407] Connection header received
0807 11:22:34.254792 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.254800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.254930 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.255055 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.256153 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55972: BlockingRecv error: recv got EOF from 127.0.0.1:55972 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
W20260807 11:22:34.275646 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.272774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.2.223.189:39543)
0807 11:22:34.272975 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.272982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.273720 (+ 738us) server_negotiation.cc:407] Connection header received
0807 11:22:34.273925 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.273931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.274106 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.274251 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.275302 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55982: BlockingRecv error: recv got EOF from 127.0.0.1:55982 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 11:22:34.301828 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.299250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.2.223.189:39543)
0807 11:22:34.299452 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.299459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.300035 (+ 576us) server_negotiation.cc:407] Connection header received
0807 11:22:34.300272 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.300278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.300402 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.300540 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.301506 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55998: BlockingRecv error: recv got EOF from 127.0.0.1:55998 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:22:34.340152 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.337724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.2.223.189:39543)
0807 11:22:34.337966 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.337972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.338417 (+ 445us) server_negotiation.cc:407] Connection header received
0807 11:22:34.338645 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.338652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.338757 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.338889 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.339841 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56004: BlockingRecv error: recv got EOF from 127.0.0.1:56004 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260807 11:22:34.410228 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.408156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.2.223.189:39543)
0807 11:22:34.408357 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.408362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.408858 (+ 496us) server_negotiation.cc:407] Connection header received
0807 11:22:34.409038 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.409042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.409139 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.409242 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.410032 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56020: BlockingRecv error: recv got EOF from 127.0.0.1:56020 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:22:34.543711 26982 catalog_manager.cc:1259] Loaded cluster ID: d15231c609e340b4964f77cdd85e7d02
I20260807 11:22:34.543852 26982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: loading cluster ID for follower catalog manager: success
W20260807 11:22:34.544873 26969 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:34.542558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.2.223.189:39543)
0807 11:22:34.542831 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.542836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.543211 (+ 375us) server_negotiation.cc:407] Connection header received
0807 11:22:34.543430 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.543434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.543525 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.543627 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:34.544610 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56032: BlockingRecv error: recv got EOF from 127.0.0.1:56032 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 11:22:34.547428 26982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: acquiring CA information for follower catalog manager: success
I20260807 11:22:34.549600 26982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b76f8a8ddaa4dce8d2a5ba2ce633acb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:22:34.618018 26999 catalog_manager.cc:1259] Loaded cluster ID: d15231c609e340b4964f77cdd85e7d02
I20260807 11:22:34.618141 26999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: loading cluster ID for follower catalog manager: success
I20260807 11:22:34.620883 26999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: acquiring CA information for follower catalog manager: success
I20260807 11:22:34.623021 26999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 886ffc4293234e8086384cd515f0e683: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:22:34.812188 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:34.805453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.2.223.189:39543)
0807 11:22:34.805703 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.805711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.806042 (+ 331us) server_negotiation.cc:407] Connection header received
0807 11:22:34.806316 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.806320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.806539 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.806684 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:34.807175 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.807929 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.809598 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.810051 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.810220 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:34.810884 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:34.811477 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:34.811665 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:34.811791 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 11:22:34.811891 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 11:22:34.830168 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:34.822301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.2.223.188:34595)
0807 11:22:34.822997 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.823006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.823475 (+ 469us) server_negotiation.cc:407] Connection header received
0807 11:22:34.823828 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.823833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.824029 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.824254 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:34.824762 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.825619 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.827321 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.827800 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.827976 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:34.828605 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:34.829317 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:34.829505 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:34.829688 (+ 183us) server_negotiation.cc:299] Negotiation successful
0807 11:22:34.829798 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":392,"threads_started":1}
I20260807 11:22:34.847103 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:34.839240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.2.223.189:39543)
0807 11:22:34.839442 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.839455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.839971 (+ 516us) server_negotiation.cc:407] Connection header received
0807 11:22:34.840219 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.840225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.840330 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.840397 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:34.841310 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.842515 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.844700 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.845295 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.845438 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:34.846058 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:34.846194 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:34.846296 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:34.846560 (+ 264us) server_negotiation.cc:299] Negotiation successful
0807 11:22:34.846658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 11:22:34.848055 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:34.838198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.2.223.190:35967)
0807 11:22:34.838974 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.838981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.839077 (+ 96us) server_negotiation.cc:407] Connection header received
0807 11:22:34.839422 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.839433 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.839770 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.840001 (+ 231us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:34.840595 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.841795 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.844172 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.844927 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.845125 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:34.845728 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:34.846223 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:34.846330 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:34.847555 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0807 11:22:34.847669 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":433,"threads_started":1}
I20260807 11:22:34.853308 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:34.840285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.2.223.188:34595)
0807 11:22:34.840495 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:34.840502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:34.841458 (+ 956us) server_negotiation.cc:407] Connection header received
0807 11:22:34.841670 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:34.841676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:34.841770 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:34.841877 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:34.842874 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.843625 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.846348 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:34.847108 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:34.847267 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:34.847778 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:34.847927 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:34.848029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:34.852659 (+ 4630us) server_negotiation.cc:299] Negotiation successful
0807 11:22:34.852864 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:22:34.854321 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26730
I20260807 11:22:34.856387 26796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:34.974931 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26730
W20260807 11:22:35.003209 26910 connection.cc:535] server connection from 127.0.0.1:41752 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 11:22:35.003293 26830 connection.cc:535] server connection from 127.0.0.1:55898 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:22:35.003782 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26809
I20260807 11:22:35.005143 26876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:35.125569 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26809
I20260807 11:22:35.154704 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 26890
I20260807 11:22:35.156080 26956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:35.273497 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 26890
2026-08-07T11:22:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5258 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:22:35 dist-test-slave-5l3k krb5kdc[27120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:22:35 dist-test-slave-5l3k krb5kdc[27120](info): set up 2 sockets
Aug 07 11:22:35 dist-test-slave-5l3k krb5kdc[27120](info): commencing operation
krb5kdc: starting...
W20260807 11:22:36.515183 2942 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.009s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/test-user.keytab.
Aug 07 11:22:36 dist-test-slave-5l3k krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.223.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
Entry for principal kudu/127.2.223.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab.
I20260807 11:22:36.951821 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46871
--webserver_interface=127.2.223.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:22:37.347677 27133 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:37.348122 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:37.348289 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:37.348392 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:37.356418 27133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:37.356554 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:37.356712 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:37.356822 27133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:37.356942 27133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:37.368218 27133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.190.keytab
--principal=kudu/127.2.223.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.190:46871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.223.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:37.369808 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:37.371537 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:37.386698 27141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:37.386796 27142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:37.386745 27145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:37.388166 27133 server_base.cc:1034] running on GCE node
I20260807 11:22:37.389158 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:37.391024 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:37.392336 27133 hybrid_clock.cc:648] HybridClock initialized: now 1786101757392276 us; error 66 us; skew 500 ppm
I20260807 11:22:37.392709 27133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:37 dist-test-slave-5l3k krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:37.401800 27133 init.cc:377] Logged in from keytab as kudu/127.2.223.190@KRBTEST.COM (short username kudu)
I20260807 11:22:37.404036 27133 webserver.cc:460] Webserver started at http://127.2.223.190:43889/ using document root <none> and password file <none>
I20260807 11:22:37.404940 27133 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:37.405081 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:37.405468 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:37.409240 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data/instance:
uuid: "1232d17ed3bb4d1293fe7727580e274e"
format_stamp: "Formatted at 2026-08-07 11:22:37 on dist-test-slave-5l3k"
server_key: "0944aa9dc82b99d3481f696887efc791"
server_key_iv: "d251ff8c10293e3c05d54daa18351834"
server_key_version: "encryptionkey@0"
I20260807 11:22:37.410300 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance:
uuid: "1232d17ed3bb4d1293fe7727580e274e"
format_stamp: "Formatted at 2026-08-07 11:22:37 on dist-test-slave-5l3k"
server_key: "0944aa9dc82b99d3481f696887efc791"
server_key_iv: "d251ff8c10293e3c05d54daa18351834"
server_key_version: "encryptionkey@0"
I20260807 11:22:37.416890 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:22:37.421415 27152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:37.423460 27133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:22:37.423730 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal
uuid: "1232d17ed3bb4d1293fe7727580e274e"
format_stamp: "Formatted at 2026-08-07 11:22:37 on dist-test-slave-5l3k"
server_key: "0944aa9dc82b99d3481f696887efc791"
server_key_iv: "d251ff8c10293e3c05d54daa18351834"
server_key_version: "encryptionkey@0"
I20260807 11:22:37.423996 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:37.447446 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:37.452567 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:37.452941 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:37.475907 27133 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.190:46871
I20260807 11:22:37.475927 27204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.190:46871 every 8 connection(s)
I20260807 11:22:37.477595 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/data/info.pb
I20260807 11:22:37.478116 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 27133
I20260807 11:22:37.478343 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-0/wal/instance
I20260807 11:22:37.479067 2942 external_mini_cluster.cc:1428] Setting key 236e80b7e201b3f962354342adc5edbb
I20260807 11:22:37.483738 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:37.495679 27205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:37 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.223.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.189@KRBTEST.COM" created.
I20260807 11:22:37.524473 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.500631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.2.223.190:46871)
0807 11:22:37.502860 (+ 2229us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:37.502873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:37.502945 (+ 72us) server_negotiation.cc:407] Connection header received
0807 11:22:37.505010 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:37.505018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:37.505390 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:37.505661 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:37.507394 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:37.508730 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:37.510372 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:37.511121 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:37.511376 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:37.515588 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:37.515599 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:37.515616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:37.515665 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:37.519217 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:37.520154 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:37.520164 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:37.520166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:37.520400 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:37.520884 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:37.520889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:37.520892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:37.521606 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:37.521863 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:37.522703 (+ 840us) server_negotiation.cc:299] Negotiation successful
0807 11:22:37.523345 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":1940,"thread_start_us":1645,"threads_started":1}
I20260807 11:22:37.524474 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.500622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46871 (local address 127.0.0.1:40756)
0807 11:22:37.501739 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.501824 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:37.504454 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:37.505679 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:37.505686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:37.506667 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:37.507097 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:37.507122 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:37.508938 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:37.508944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:37.510181 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:37.510198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:37.511426 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:37.511434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:37.511613 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:37.512422 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:37.512450 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:37.515267 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:37.519420 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:37.519436 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:37.519452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:37.519965 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:37.520555 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:37.520560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:37.520564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:37.520754 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:37.521772 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:37.521781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:37.522042 (+ 261us) client_negotiation.cc:769] Sending connection context
0807 11:22:37.522399 (+ 357us) client_negotiation.cc:240] Negotiation successful
0807 11:22:37.523050 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":482,"threads_started":1}
I20260807 11:22:37.528887 27205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:37.530385 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.529773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46412)
0807 11:22:37.530013 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.530190 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260807 11:22:37.530710 27155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
I20260807 11:22:37.532884 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.532290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46424)
0807 11:22:37.532507 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.532647 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 11:22:37.533174 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:40115: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
Entry for principal kudu/127.2.223.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab.
I20260807 11:22:37.569917 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:40115
--webserver_interface=127.2.223.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:22:37.582682 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } attempt: 1
I20260807 11:22:37.584264 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.583595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46430)
0807 11:22:37.583884 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.584050 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 11:22:37.585829 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.585258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46442)
0807 11:22:37.585471 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.585588 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 11:22:37.586138 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:40115: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
I20260807 11:22:37.654429 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } attempt: 2
I20260807 11:22:37.655910 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.655351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46448)
0807 11:22:37.655631 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.655746 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 11:22:37.657191 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.656737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46458)
0807 11:22:37.656914 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.657020 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:22:37.657416 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:40115: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
I20260807 11:22:37.748718 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } attempt: 3
I20260807 11:22:37.749946 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.749442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46462)
0807 11:22:37.749685 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.749795 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 11:22:37.751214 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.750771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46464)
0807 11:22:37.750949 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.751039 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 11:22:37.751428 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:40115: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
I20260807 11:22:37.894690 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } attempt: 4
I20260807 11:22:37.896250 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.895629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46478)
0807 11:22:37.895986 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.896089 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260807 11:22:37.897429 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:37.897028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46482)
0807 11:22:37.897191 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:37.897290 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:22:37.897635 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.189:40115: Network error: Client connection negotiation failed: client connection to 127.2.223.189:40115: connect: Connection refused (error 111)
W20260807 11:22:37.984535 27212 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:37.985018 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:37.985162 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:37.985307 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:37.993726 27212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:37.993892 27212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:37.994031 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:37.994127 27212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:37.994243 27212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:38.005911 27212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.189.keytab
--principal=kudu/127.2.223.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.189:40115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.223.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:38.007565 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:38.009137 27212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:22:38.023171 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:38.023244 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:38.023267 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:38.024673 27212 server_base.cc:1034] running on GCE node
I20260807 11:22:38.026064 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:38.028213 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:38.029529 27212 hybrid_clock.cc:648] HybridClock initialized: now 1786101758029441 us; error 49 us; skew 500 ppm
I20260807 11:22:38.029965 27212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:38.033850 27212 init.cc:377] Logged in from keytab as kudu/127.2.223.189@KRBTEST.COM (short username kudu)
I20260807 11:22:38.035672 27212 webserver.cc:460] Webserver started at http://127.2.223.189:34571/ using document root <none> and password file <none>
I20260807 11:22:38.036465 27212 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:38.036568 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:38.036871 27212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:38.040437 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data/instance:
uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "6c8760d46dd03cf0aa4af2404bbb34b5"
server_key_iv: "d40ca5595b2afe5731a29621d601aa31"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.041332 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal/instance:
uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "6c8760d46dd03cf0aa4af2404bbb34b5"
server_key_iv: "d40ca5595b2afe5731a29621d601aa31"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.047542 27212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:22:38.051656 27228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:38.053357 27212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:22:38.053598 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal
uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "6c8760d46dd03cf0aa4af2404bbb34b5"
server_key_iv: "d40ca5595b2afe5731a29621d601aa31"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.053853 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:38.073354 27212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:38.078243 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:38.078639 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:38.101174 27212 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.189:40115
I20260807 11:22:38.101197 27281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.189:40115 every 8 connection(s)
I20260807 11:22:38.102946 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/data/info.pb
I20260807 11:22:38.104594 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 27212
I20260807 11:22:38.104816 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-1/wal/instance
I20260807 11:22:38.105422 2942 external_mini_cluster.cc:1428] Setting key 46ad4afe47fa16da8060d86a61911e9f
I20260807 11:22:38.108623 27282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:38.120136 27282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:38.148430 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.124369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.2.223.190:46871)
0807 11:22:38.124975 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.124985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.126342 (+ 1357us) server_negotiation.cc:407] Connection header received
0807 11:22:38.128717 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.128726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.128841 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.129038 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.131136 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.132331 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.134146 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.134967 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.135155 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.139629 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.139640 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.139648 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.139724 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.144098 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.145261 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.145272 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.145278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.145495 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.146027 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.146035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.146041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.146547 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.146728 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.147603 (+ 875us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.147910 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.2.223.188@KRBTEST.COM; defaulting to no policy
I20260807 11:22:38.149226 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.125064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46871 (local address 127.0.0.1:40770)
0807 11:22:38.126075 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.126141 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.128340 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.129361 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.129372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.130017 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.130799 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.130825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.132559 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.132562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.133918 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.133935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.135254 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.135260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.135502 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.136360 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.136388 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.139260 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.144402 (+ 5142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.144411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.144430 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.145031 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.145641 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.145645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.145657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.145874 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.146702 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.146710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.147135 (+ 425us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.147405 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.147881 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":461,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.223.188@KRBTEST.COM" created.
I20260807 11:22:38.151561 27282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:38.170150 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.152307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46496)
0807 11:22:38.152504 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.152529 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.152648 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.154009 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.154013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.154600 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.154756 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.154766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.156242 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.156250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.157164 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.157177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.158178 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.158183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.158389 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.159220 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.159238 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.161920 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.167179 (+ 5259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.167184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.167187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.167623 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.168155 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.168158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.168160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.168318 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.169210 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.169213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.169342 (+ 129us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.169462 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.169715 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 11:22:38.170583 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.152773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.2.223.189:40115)
0807 11:22:38.153511 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.153519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.153551 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:22:38.153608 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.153614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.153841 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.154022 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.154954 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.156082 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.157319 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.158015 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.158155 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.162302 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.162325 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.162350 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.162401 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.166899 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.167805 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.167817 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.167824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.168002 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.168441 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.168449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.168455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.169036 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.169283 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.169700 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.170039 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":419,"threads_started":1}
I20260807 11:22:38.173166 27282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:38.174696 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.173959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57776)
0807 11:22:38.174250 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.174421 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 11:22:38.174989 27230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.176964 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.176436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57792)
0807 11:22:38.176653 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.176780 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 11:22:38.177230 27282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
Entry for principal kudu/127.2.223.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab.
I20260807 11:22:38.196934 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } attempt: 5
I20260807 11:22:38.198683 2942 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
/tmp/dist-test-taskgCuO6z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:38451
--webserver_interface=127.2.223.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.223.148:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:38.214313 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.198170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.2.223.189:40115)
0807 11:22:38.198426 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.198433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.198457 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:22:38.198569 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.198574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.198689 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.198823 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.199760 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.200943 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.202278 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.203150 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.203326 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.207460 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.207466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.207470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.207513 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.210661 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.211905 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.211910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.211913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.212083 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.212689 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.212695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.212697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.213073 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.213220 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.213812 (+ 592us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.214043 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 11:22:38.214665 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.197890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46510)
0807 11:22:38.198137 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.198163 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.198322 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.198881 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.198885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.199352 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.199555 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.199576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.201151 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.201155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.202078 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.202094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.203366 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.203371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.203568 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.204314 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.204336 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.207159 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.210927 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.210937 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.210944 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.211691 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.212257 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.212266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.212272 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.212499 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.213266 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.213274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.213500 (+ 226us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.213687 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.214133 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 11:22:38.216289 27205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:22:38.218173 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.217521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57798)
0807 11:22:38.217802 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.217936 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260807 11:22:38.219615 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.219214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57808)
0807 11:22:38.219375 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.219461 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:22:38.219926 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.226694 27282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 1
I20260807 11:22:38.228138 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.227524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57812)
0807 11:22:38.227747 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.227879 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:22:38.229569 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.228994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57818)
0807 11:22:38.229284 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.229405 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260807 11:22:38.229832 27282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.271209 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 1
I20260807 11:22:38.272657 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.272116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57830)
0807 11:22:38.272359 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.272460 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 11:22:38.273859 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.273464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57834)
0807 11:22:38.273639 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.273730 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 11:22:38.274108 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.298089 27282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 2
I20260807 11:22:38.299458 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.298926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57836)
0807 11:22:38.299175 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.299281 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 11:22:38.300681 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.300282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57844)
0807 11:22:38.300459 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.300537 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 11:22:38.300936 27282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.342392 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 2
I20260807 11:22:38.343801 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.343225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57858)
0807 11:22:38.343490 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.343604 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 11:22:38.344916 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.344529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57860)
0807 11:22:38.344697 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.344783 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:22:38.345141 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.392249 27282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 3
I20260807 11:22:38.393532 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.392990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57876)
0807 11:22:38.393256 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.393358 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 11:22:38.394810 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.394242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57878)
0807 11:22:38.394577 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.394674 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260807 11:22:38.395056 27282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.451414 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 3
I20260807 11:22:38.452817 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.452265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57882)
0807 11:22:38.452501 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.452616 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 11:22:38.454049 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.453583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57890)
0807 11:22:38.453830 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.453917 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260807 11:22:38.454279 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
I20260807 11:22:38.538300 27282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 4
I20260807 11:22:38.539659 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.539106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57896)
0807 11:22:38.539370 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.539474 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:22:38.540747 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.540383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57906)
0807 11:22:38.540538 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.540629 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 11:22:38.540968 27282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
W20260807 11:22:38.607380 27291 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:22:38.607856 27291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:22:38.607995 27291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:22:38.608155 27291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:22:38.616047 27291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:22:38.616225 27291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:22:38.616346 27291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:22:38.616444 27291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:22:38.616550 27291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:22:38.627689 27291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.223.148:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.223.190:46871,127.2.223.189:40115,127.2.223.188:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/krb5kdc/kudu_127.2.223.188.keytab
--principal=kudu/127.2.223.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.223.188:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.223.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:22:38.629455 27291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:22:38.631178 27291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 11:22:38.631532 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 4
I20260807 11:22:38.632766 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.632274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57922)
0807 11:22:38.632485 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.632593 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 11:22:38.634001 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.633519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57926)
0807 11:22:38.633699 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.633819 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 11:22:38.634275 27205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.223.188:38451: Network error: Client connection negotiation failed: client connection to 127.2.223.188:38451: connect: Connection refused (error 111)
W20260807 11:22:38.646011 27299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:38.646098 27296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:22:38.646075 27297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:22:38.647033 27291 server_base.cc:1034] running on GCE node
I20260807 11:22:38.647953 27291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:22:38.649842 27291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:22:38.651167 27291 hybrid_clock.cc:648] HybridClock initialized: now 1786101758651073 us; error 78 us; skew 500 ppm
I20260807 11:22:38.651641 27291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:22:38.655620 27291 init.cc:377] Logged in from keytab as kudu/127.2.223.188@KRBTEST.COM (short username kudu)
I20260807 11:22:38.657518 27291 webserver.cc:460] Webserver started at http://127.2.223.188:41907/ using document root <none> and password file <none>
I20260807 11:22:38.658622 27291 fs_manager.cc:359] Metadata directory not provided
I20260807 11:22:38.658771 27291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:22:38.659158 27291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:22:38.662977 27291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data/instance:
uuid: "f7ac328a658b4c7b958b97b9f4ff40fe"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "0896536f6aa7ceee05723a5f26b39b47"
server_key_iv: "f85cea7b8e381da89d09d51e0da953f0"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.664139 27291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal/instance:
uuid: "f7ac328a658b4c7b958b97b9f4ff40fe"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "0896536f6aa7ceee05723a5f26b39b47"
server_key_iv: "f85cea7b8e381da89d09d51e0da953f0"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.670776 27291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 11:22:38.675191 27307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:22:38.676949 27291 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:22:38.677208 27291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data,/tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal
uuid: "f7ac328a658b4c7b958b97b9f4ff40fe"
format_stamp: "Formatted at 2026-08-07 11:22:38 on dist-test-slave-5l3k"
server_key: "0896536f6aa7ceee05723a5f26b39b47"
server_key_iv: "f85cea7b8e381da89d09d51e0da953f0"
server_key_version: "encryptionkey@0"
I20260807 11:22:38.677489 27291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:22:38.712849 27291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:22:38.718149 27291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:22:38.718546 27291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:22:38.739768 27291 rpc_server.cc:307] RPC server started. Bound to: 127.2.223.188:38451
I20260807 11:22:38.739774 27363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.223.188:38451 every 8 connection(s)
I20260807 11:22:38.741398 27291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/data/info.pb
I20260807 11:22:38.745676 27364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:22:38.746466 2942 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu as pid 27291
I20260807 11:22:38.746711 2942 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgCuO6z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786101369622807-2942-0/minicluster-data/master-2/wal/instance
I20260807 11:22:38.747301 2942 external_mini_cluster.cc:1428] Setting key 22bc7945408de4c42f5810750c99b16d
I20260807 11:22:38.756775 27364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:38.775724 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.749531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.2.223.188:38451)
0807 11:22:38.750708 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.750722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.750772 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:22:38.753036 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.753067 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.753624 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.754052 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.756251 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.758424 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.760224 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.761559 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.761804 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.767085 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.767103 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.767134 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.767191 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.770769 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.771655 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.771664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.771673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.771865 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.772338 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.772346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.772354 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.772996 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.773343 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.773914 (+ 571us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.774415 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":429,"threads_started":1}
I20260807 11:22:38.776924 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.760082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.2.223.190:46871)
0807 11:22:38.760719 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.760727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.761226 (+ 499us) server_negotiation.cc:407] Connection header received
0807 11:22:38.761479 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.761484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.761551 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.761683 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.762957 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.763773 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.765291 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.765847 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.765974 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.770366 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.770372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.770376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.770428 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.773545 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.774430 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.774441 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.774444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.774675 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.775120 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.775125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.775128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.775437 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.775626 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.776230 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.776491 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":301,"threads_started":1}
I20260807 11:22:38.776948 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.760121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.190:46871 (local address 127.0.0.1:40776)
0807 11:22:38.761067 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.761124 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.761324 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.761761 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.761768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.762464 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.762800 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.762810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.763945 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.763950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.765095 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.765127 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.766137 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.766149 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.766368 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.767250 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.767269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.770074 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.773721 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.773735 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.773744 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.774298 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.774811 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.774816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.774820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.774995 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.775596 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.775605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.775834 (+ 229us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.776108 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.776489 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":520,"threads_started":1}
I20260807 11:22:38.779282 27364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.190@KRBTEST.COM
I20260807 11:22:38.796432 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.780090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.189:40115 (local address 127.0.0.1:46512)
0807 11:22:38.780354 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.780368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.780478 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.781140 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.781145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.781680 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.781903 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.781915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.783471 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.783476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.784712 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.784729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.785781 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.785785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.786056 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.786778 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.786806 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.789685 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.793415 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.793421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.793425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.794019 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.794434 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.794439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.794442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.794695 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.795340 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.795344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.795558 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.795736 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.796032 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260807 11:22:38.796490 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.780262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.2.223.189:40115)
0807 11:22:38.780840 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.780844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.780860 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:22:38.780910 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.780913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.780971 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.781105 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.782119 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.783306 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.784905 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.785593 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.785759 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.789979 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.789989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.789995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.790060 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.793228 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.794158 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.794162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.794164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.794308 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.794822 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.794831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.794837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.795213 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.795380 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.795887 (+ 507us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.796154 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":343,"threads_started":1}
I20260807 11:22:38.797987 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.779528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.2.223.190:46871)
0807 11:22:38.779723 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.779728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.779743 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:22:38.779831 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.779836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.779941 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.780064 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.781749 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.782704 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.784357 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.784922 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.785132 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.791129 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.791138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.791145 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.791199 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.794487 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.795500 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.795507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.795511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.795685 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.796139 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.796144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.796148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.796521 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.796839 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.797380 (+ 541us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.797601 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:22:38.798171 27364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.188@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.223.189@KRBTEST.COM
I20260807 11:22:38.815471 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.799398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.2.223.188:38451)
0807 11:22:38.799561 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.799566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.799584 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:22:38.800221 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.800229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.800310 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.800408 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.801284 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.802334 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.803907 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.804595 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.804742 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.808946 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.808951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.808954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.808990 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.812194 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.813192 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.813197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.813213 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.813350 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.813817 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.813821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.813823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.814143 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.814285 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.814988 (+ 703us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.815192 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:22:38.815896 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.799253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57946)
0807 11:22:38.799447 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.799469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.799585 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.800460 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.800465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.800961 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.801118 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.801127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.802472 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.802506 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.803751 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.803765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.804767 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.804771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.805045 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.806084 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.806102 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.808717 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.812413 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.812419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.812423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.813035 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.813487 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.813493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.813497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.813698 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.814461 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.814469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.814711 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.814904 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.815433 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":79}
I20260807 11:22:38.818557 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.800062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.2.223.189:40115)
0807 11:22:38.800248 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.800253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.800270 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:22:38.800408 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.800414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.800496 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.800601 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.802146 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.803246 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.804938 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.805549 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.805657 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.811119 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.811125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.811128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.811170 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.814923 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.816030 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.816040 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.816046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.816244 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.816756 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.816760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.816762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.817067 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.817370 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.817790 (+ 420us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.818082 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:22:38.822163 27364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: Bootstrap starting.
I20260807 11:22:38.824358 2942 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:22:38.826282 27364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:38.827963 27364 log.cc:824] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:38.827306 27179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1232d17ed3bb4d1293fe7727580e274e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790
I20260807 11:22:38.832333 27364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: No bootstrap required, opened a new log
I20260807 11:22:38.840173 27282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 5
I20260807 11:22:38.844878 27364 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.845467 27364 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac328a658b4c7b958b97b9f4ff40fe, State: Initialized, Role: FOLLOWER
I20260807 11:22:38.846037 27364 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.847334 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:38.847828 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:38.849009 27364 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: configured and running, proceeding with master startup.
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101758, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.189@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:22:38.860291 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.841026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57952)
0807 11:22:38.841282 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.841306 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.841469 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.841884 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.841889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.842341 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.842589 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.842602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.843772 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.843778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.844699 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.844713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.845729 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.845737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.846001 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.846820 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.846854 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.850291 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.855861 (+ 5570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.855868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.855875 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.856719 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.857456 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.857463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.857467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.857701 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.859021 (+ 1320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.859027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.859231 (+ 204us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.859402 (+ 171us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.859782 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 11:22:38.860364 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.841141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.2.223.188:38451)
0807 11:22:38.841375 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.841382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.841406 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:22:38.841633 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.841638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.841713 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.841859 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.842772 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.843585 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.844893 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.845459 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.845634 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.850655 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.850664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.850669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.850723 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.855548 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.856975 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.856983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.856986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.857233 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.857907 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.857916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.857923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.858760 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.859030 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.859576 (+ 546us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.859864 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:22:38.867574 27282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: Bootstrap starting.
W20260807 11:22:38.869207 27383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:22:38.869385 27383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:22:38.873700 27282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:38.875355 27282 log.cc:824] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:38.880147 27282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: No bootstrap required, opened a new log
I20260807 11:22:38.871501 27291 master_runner.cc:186] Error getting master registration for 127.2.223.190:46871: OK, instance_id { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" instance_seqno: 1786101757469455 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:22:38.883301 27291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:22:38.896797 27282 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.897395 27282 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8b23236d6f4d5cb35ba03fbd92c10a, State: Initialized, Role: FOLLOWER
I20260807 11:22:38.898173 27282 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.899425 27386 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:38.899880 27386 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:38.901070 27282 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:22:38.911540 27205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } attempt: 5
W20260807 11:22:38.917255 27399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:22:38.917524 27399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 07 11:22:38 dist-test-slave-5l3k krb5kdc[27120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786101757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.223.190@KRBTEST.COM for kudu/127.2.223.188@KRBTEST.COM
I20260807 11:22:38.916428 27212 master_runner.cc:186] Error getting master registration for 127.2.223.190:46871: OK, instance_id { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" instance_seqno: 1786101757469455 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:22:38.934499 27212 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:22:38.941749 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.912848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.2.223.188:38451)
0807 11:22:38.913353 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:38.913359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:38.913382 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:22:38.913450 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:38.913458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:38.913546 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:38.913671 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:38.915018 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.916183 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.928617 (+ 12434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.929441 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.929606 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.934192 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:38.934201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:38.934205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:38.934254 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:38.938412 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.939326 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.939333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.939341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.939544 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:38.940042 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:38.940048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:38.940052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:38.940471 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:38.940636 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:38.941073 (+ 437us) server_negotiation.cc:299] Negotiation successful
0807 11:22:38.941329 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260807 11:22:38.941769 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:38.912350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.223.188:38451 (local address 127.0.0.1:57962)
0807 11:22:38.912712 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 11:22:38.912734 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:22:38.912866 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:22:38.913899 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:22:38.913905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:22:38.914587 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:22:38.914818 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.914833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.926635 (+ 11802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.926644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.928357 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:22:38.928385 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:22:38.929639 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:22:38.929647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:22:38.929880 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:38.930769 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:22:38.930799 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:22:38.933926 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:22:38.938645 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.938651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.938656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.939123 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.939680 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:22:38.939686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:22:38.939690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:22:38.939893 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:22:38.940601 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:22:38.940606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:22:38.940785 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 11:22:38.940944 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 11:22:38.941307 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260807 11:22:38.947892 27205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e: Bootstrap starting.
I20260807 11:22:38.952507 27205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e: Neither blocks nor log segments found. Creating new log.
I20260807 11:22:38.954109 27205 log.cc:824] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:22:38.958331 27205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e: No bootstrap required, opened a new log
I20260807 11:22:38.966960 27205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.967444 27205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1232d17ed3bb4d1293fe7727580e274e, State: Initialized, Role: FOLLOWER
I20260807 11:22:38.968016 27205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.969496 27402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:38.970299 27402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:38.970914 27205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:22:38.970935 27314 tablet.cc:2369] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:22:38.971946 27179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:22:38.972152 27179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:38.976908 27179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:38.980140 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1232d17ed3bb4d1293fe7727580e274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a"
I20260807 11:22:38.980738 27256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:38.980950 27337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1232d17ed3bb4d1293fe7727580e274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe"
I20260807 11:22:38.981590 27337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:22:38.984690 27256 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1232d17ed3bb4d1293fe7727580e274e in term 1.
I20260807 11:22:38.985841 27155 leader_election.cc:302] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1232d17ed3bb4d1293fe7727580e274e, cd8b23236d6f4d5cb35ba03fbd92c10a; no voters:
I20260807 11:22:38.986099 27337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1232d17ed3bb4d1293fe7727580e274e in term 1.
I20260807 11:22:38.987591 27179 leader_election.cc:288] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [CANDIDATE]: Term 1 election: Requested vote from peers cd8b23236d6f4d5cb35ba03fbd92c10a (127.2.223.189:40115), f7ac328a658b4c7b958b97b9f4ff40fe (127.2.223.188:38451)
I20260807 11:22:38.995014 27402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:22:39.003181 27402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [term 1 LEADER]: Becoming Leader. State: Replica: 1232d17ed3bb4d1293fe7727580e274e, State: Running, Role: LEADER
I20260807 11:22:39.004542 27402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } }
I20260807 11:22:39.024042 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1232d17ed3bb4d1293fe7727580e274e. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.024740 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:39.032645 27416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:22:39.052115 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:39.017888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.2.223.188:38451)
0807 11:22:39.018199 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.018209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.021253 (+ 3044us) server_negotiation.cc:407] Connection header received
0807 11:22:39.021610 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.021617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.021715 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.021813 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.027046 (+ 5233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.028442 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.032658 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.033375 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.033489 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:39.042564 (+ 9075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:39.042573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:39.042578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:39.042628 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:39.046683 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.048324 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.048331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.048335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.048526 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.049387 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.049393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.049396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.049810 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:39.050130 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:39.051417 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0807 11:22:39.051737 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:22:39.059358 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:39.019680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.2.223.190:46871)
0807 11:22:39.019936 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.019945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.030275 (+ 10330us) server_negotiation.cc:407] Connection header received
0807 11:22:39.030634 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.030641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.030751 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.030844 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.033306 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.034563 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.040608 (+ 6045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.042097 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.042227 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:39.051116 (+ 8889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:39.051125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:39.051130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:39.051184 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:39.055325 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.056385 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.056392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.056397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.056610 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.057117 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.057122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.057126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.057619 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:39.057976 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:39.058605 (+ 629us) server_negotiation.cc:299] Negotiation successful
0807 11:22:39.058907 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:22:39.067209 27416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260807 11:22:39.081883 27235 tablet.cc:2369] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:22:39.086892 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:39.021411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.2.223.189:40115)
0807 11:22:39.021640 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.021646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.035768 (+ 14122us) server_negotiation.cc:407] Connection header received
0807 11:22:39.038970 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.038977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.039080 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.039167 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.042618 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.043863 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.049255 (+ 5392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:39.050086 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:39.050236 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:39.071993 (+ 21757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:22:39.072006 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:22:39.072015 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:22:39.072093 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:22:39.077883 (+ 5790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.082416 (+ 4533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.082424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.082427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.082864 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:22:39.083721 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:22:39.083730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:22:39.083741 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:22:39.084353 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:22:39.084758 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:39.085733 (+ 975us) server_negotiation.cc:299] Negotiation successful
0807 11:22:39.086207 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:22:39.101248 27427 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:22:39.120781 27337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [term 1 FOLLOWER]: Refusing update from remote peer 1232d17ed3bb4d1293fe7727580e274e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:22:39.122503 27408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:22:39.125118 27256 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [term 1 FOLLOWER]: Refusing update from remote peer 1232d17ed3bb4d1293fe7727580e274e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:22:39.126843 27408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:22:39.148825 27386 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1232d17ed3bb4d1293fe7727580e274e. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.149156 27386 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:39.153802 27425 mvcc.cc:205] Tried to move back new op lower bound from 7315872805342482432 to 7315872804913827840. Current Snapshot: MvccSnapshot[applied={T|T < 7315872805342482432}]
I20260807 11:22:39.156208 27429 mvcc.cc:205] Tried to move back new op lower bound from 7315872805342482432 to 7315872804913827840. Current Snapshot: MvccSnapshot[applied={T|T < 7315872805342482432}]
I20260807 11:22:39.171573 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1232d17ed3bb4d1293fe7727580e274e. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.171974 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:39.178565 27402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.179112 27402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:39.194645 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.195324 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e [sys.catalog]: This master's current role is: LEADER
I20260807 11:22:39.205529 27416 catalog_manager.cc:1347] Generated new cluster ID: 7bb2ff4281fc4ba28e037132156eccad
I20260807 11:22:39.205710 27416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:22:39.206964 27386 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.207296 27386 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:39.215461 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1232d17ed3bb4d1293fe7727580e274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232d17ed3bb4d1293fe7727580e274e" member_type: VOTER last_known_addr { host: "127.2.223.190" port: 46871 } } peers { permanent_uuid: "cd8b23236d6f4d5cb35ba03fbd92c10a" member_type: VOTER last_known_addr { host: "127.2.223.189" port: 40115 } } peers { permanent_uuid: "f7ac328a658b4c7b958b97b9f4ff40fe" member_type: VOTER last_known_addr { host: "127.2.223.188" port: 38451 } } }
I20260807 11:22:39.215788 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:22:39.258406 27416 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:22:39.267750 27416 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:22:39.293370 27416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1232d17ed3bb4d1293fe7727580e274e: Generated new TSK 0
I20260807 11:22:39.296264 27416 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:22:39.475478 27368 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.453211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.2.223.190:46871)
0807 11:22:39.453513 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.453521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.472534 (+ 19013us) server_negotiation.cc:407] Connection header received
0807 11:22:39.473610 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.473617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.473732 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.473820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.475198 (+ 1378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40800: BlockingRecv error: recv got EOF from 127.0.0.1:40800 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:22:39.493476 27367 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.454810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.2.223.188:38451)
0807 11:22:39.455014 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.455022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.491108 (+ 36086us) server_negotiation.cc:407] Connection header received
0807 11:22:39.491817 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.491824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.491920 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.491995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.493152 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57974: BlockingRecv error: recv got EOF from 127.0.0.1:57974 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 11:22:39.498150 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.494861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.2.223.189:40115)
0807 11:22:39.495101 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.495107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.495895 (+ 788us) server_negotiation.cc:407] Connection header received
0807 11:22:39.496232 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.496239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.496329 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.496413 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.497849 (+ 1436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46546: BlockingRecv error: recv got EOF from 127.0.0.1:46546 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:22:39.542138 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.531892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.2.223.189:40115)
0807 11:22:39.532135 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.532142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.539507 (+ 7365us) server_negotiation.cc:407] Connection header received
0807 11:22:39.540177 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.540184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.540350 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.540439 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.541849 (+ 1410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46548: BlockingRecv error: recv got EOF from 127.0.0.1:46548 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 11:22:39.585671 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.576896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.2.223.189:40115)
0807 11:22:39.577130 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.577139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.583795 (+ 6656us) server_negotiation.cc:407] Connection header received
0807 11:22:39.584044 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.584051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.584164 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.584304 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.585360 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46562: BlockingRecv error: recv got EOF from 127.0.0.1:46562 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260807 11:22:39.594753 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.592472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.2.223.189:40115)
0807 11:22:39.592722 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.592731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.592879 (+ 148us) server_negotiation.cc:407] Connection header received
0807 11:22:39.593208 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.593215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.593325 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.593456 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.594443 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46576: BlockingRecv error: recv got EOF from 127.0.0.1:46576 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260807 11:22:39.605748 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.603125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.2.223.189:40115)
0807 11:22:39.603347 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.603354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.603989 (+ 635us) server_negotiation.cc:407] Connection header received
0807 11:22:39.604233 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.604242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.604354 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.604486 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.605476 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46582: BlockingRecv error: recv got EOF from 127.0.0.1:46582 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260807 11:22:39.620702 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.618071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.2.223.189:40115)
0807 11:22:39.618279 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.618285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.618956 (+ 671us) server_negotiation.cc:407] Connection header received
0807 11:22:39.619178 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.619184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.619312 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.619447 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.620414 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46588: BlockingRecv error: recv got EOF from 127.0.0.1:46588 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 11:22:39.644608 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.642409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.2.223.189:40115)
0807 11:22:39.642665 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.642672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.643105 (+ 433us) server_negotiation.cc:407] Connection header received
0807 11:22:39.643274 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.643283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.643390 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.643535 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.644349 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46590: BlockingRecv error: recv got EOF from 127.0.0.1:46590 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260807 11:22:39.683650 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.680998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.2.223.189:40115)
0807 11:22:39.681311 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.681318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.681764 (+ 446us) server_negotiation.cc:407] Connection header received
0807 11:22:39.682021 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.682026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.682154 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.682328 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.683397 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46596: BlockingRecv error: recv got EOF from 127.0.0.1:46596 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260807 11:22:39.754244 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.752124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.2.223.189:40115)
0807 11:22:39.752354 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.752361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.752734 (+ 373us) server_negotiation.cc:407] Connection header received
0807 11:22:39.752922 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.752928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.753040 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.753181 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.753983 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 11:22:39.871656 27383 catalog_manager.cc:1259] Loaded cluster ID: 7bb2ff4281fc4ba28e037132156eccad
I20260807 11:22:39.871825 27383 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: loading cluster ID for follower catalog manager: success
I20260807 11:22:39.874581 27383 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: acquiring CA information for follower catalog manager: success
I20260807 11:22:39.876679 27383 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7ac328a658b4c7b958b97b9f4ff40fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 11:22:39.888792 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:22:39.886334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.2.223.189:40115)
0807 11:22:39.886642 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:39.886650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:39.887088 (+ 438us) server_negotiation.cc:407] Connection header received
0807 11:22:39.887348 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:39.887353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:39.887454 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:39.887575 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:22:39.888470 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46620: BlockingRecv error: recv got EOF from 127.0.0.1:46620 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:22:39.919883 27399 catalog_manager.cc:1259] Loaded cluster ID: 7bb2ff4281fc4ba28e037132156eccad
I20260807 11:22:39.920006 27399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: loading cluster ID for follower catalog manager: success
I20260807 11:22:39.922744 27399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: acquiring CA information for follower catalog manager: success
I20260807 11:22:39.924857 27399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd8b23236d6f4d5cb35ba03fbd92c10a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:22:40.155896 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:40.149685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.2.223.189:40115)
0807 11:22:40.149952 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:40.149959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:40.150208 (+ 249us) server_negotiation.cc:407] Connection header received
0807 11:22:40.150388 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:40.150392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:40.150659 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:40.150773 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:40.151223 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.151945 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.153382 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.153842 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.153956 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:40.154603 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:40.155217 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:40.155343 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:40.155465 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 11:22:40.155559 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 11:22:40.175100 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:40.163038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.2.223.188:38451)
0807 11:22:40.168824 (+ 5786us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:40.168829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:40.168856 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:22:40.168979 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:40.168984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:40.169199 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:40.169457 (+ 258us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:40.170049 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.170924 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.172454 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.172878 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.173045 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:40.173667 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:40.174266 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:40.174463 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:40.174608 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:22:40.174711 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5631,"thread_start_us":5493,"threads_started":1}
I20260807 11:22:40.191877 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:40.182380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.2.223.188:38451)
0807 11:22:40.182703 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:40.182715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:40.184088 (+ 1373us) server_negotiation.cc:407] Connection header received
0807 11:22:40.184435 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:40.184444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:40.184585 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:40.184708 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:40.185946 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.187211 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.189361 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.190080 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.190230 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:40.190887 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:40.191064 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:40.191179 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:40.191392 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 11:22:40.191514 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260807 11:22:40.192215 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:40.182060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.2.223.190:46871)
0807 11:22:40.183313 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:40.183324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:40.183835 (+ 511us) server_negotiation.cc:407] Connection header received
0807 11:22:40.184132 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:40.184139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:40.184410 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:40.184673 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:40.185641 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.186745 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.189072 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.189773 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.189932 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:40.190923 (+ 991us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:40.191437 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:40.191514 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:40.191775 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 11:22:40.191884 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":837,"threads_started":1}
I20260807 11:22:40.193414 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:22:40.183943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.2.223.189:40115)
0807 11:22:40.184149 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:22:40.184154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:22:40.185767 (+ 1613us) server_negotiation.cc:407] Connection header received
0807 11:22:40.186088 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:22:40.186095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:22:40.186194 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:22:40.186308 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:22:40.187097 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.188292 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.190723 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:22:40.191534 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:22:40.191684 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:22:40.192468 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:22:40.192634 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:22:40.192713 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 11:22:40.192969 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 11:22:40.193065 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:22:40.198664 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 27133
I20260807 11:22:40.200047 27200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:40.317607 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 27133
W20260807 11:22:40.345184 27308 connection.cc:535] server connection from 127.0.0.1:57962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:22:40.345643 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 27212
I20260807 11:22:40.347016 27277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:40.464893 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 27212
I20260807 11:22:40.493165 2942 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu pid 27291
I20260807 11:22:40.494647 27359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:22:40.610445 2942 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgCuO6z/build/asan/bin/kudu with pid 27291
2026-08-07T11:22:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5333 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10592 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (390908 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST